May 25 11:14:52.556441 ip-10-0-141-158 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 11:14:52.556453 ip-10-0-141-158 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 11:14:52.556462 ip-10-0-141-158 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 11:14:52.556794 ip-10-0-141-158 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 11:15:02.583314 ip-10-0-141-158 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 25 11:15:02.583333 ip-10-0-141-158 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 89eb2f2518c14d5980402838e8ad4bff --
May 25 11:17:11.879040 ip-10-0-141-158 systemd[1]: Starting Kubernetes Kubelet...
May 25 11:17:12.354745 ip-10-0-141-158 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:12.354745 ip-10-0-141-158 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 11:17:12.354745 ip-10-0-141-158 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:12.354745 ip-10-0-141-158 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 11:17:12.354745 ip-10-0-141-158 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:12.358056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.357967    2569 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 11:17:12.365201 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365183    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:12.365201 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365198    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:12.365201 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365202    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365206    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365209    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365212    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365215    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365218    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365221    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365223    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365226    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365229    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365231    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365234    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365239    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365242    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365245    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365247    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365250    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365253    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365255    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365258    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:12.365288 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365260    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365263    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365266    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365269    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365271    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365274    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365276    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365278    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365281    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365283    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365286    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365288    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365291    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365294    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365297    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365300    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365303    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365305    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365308    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365310    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:12.365756 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365313    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365315    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365318    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365321    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365323    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365326    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365329    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365332    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365334    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365337    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365339    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365342    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365344    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365346    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365350    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365353    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365355    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365358    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365360    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365363    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:12.366267 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365365    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365368    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365370    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365373    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365375    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365378    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365380    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365384    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365389    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365392    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365394    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365398    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365400    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365403    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365406    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365408    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365410    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365413    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365417    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:12.366747 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365421    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365424    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365427    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365429    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365432    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365797    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365803    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365808    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365811    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365814    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365817    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365819    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365822    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365826    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365829    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365832    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365835    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365838    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365840    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:12.367227 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365843    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365845    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365848    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365850    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365853    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365856    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365859    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365861    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365864    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365866    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365869    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365872    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365875    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365877    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365880    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365883    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365885    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365887    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365890    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:12.367714 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365892    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365895    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365898    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365901    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365904    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365906    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365909    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365911    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365914    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365917    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365919    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365922    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365924    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365926    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365929    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365931    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365934    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365936    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365939    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365941    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:12.368200 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365944    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365946    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365949    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365951    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365954    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365957    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365960    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365962    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365964    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365967    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365969    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365972    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365974    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365978    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365982    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365985    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365988    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365991    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365994    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.365997    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:12.368699 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366000    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366002    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366005    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366007    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366010    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366012    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366014    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366017    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366019    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366022    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366024    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366027    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.366029    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366943    2569 flags.go:64] FLAG: --address="0.0.0.0"
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366952    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366959    2569 flags.go:64] FLAG: --anonymous-auth="true"
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366964    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366969    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366972    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366977    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366982    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 11:17:12.369404 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366985    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366988    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366992    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366995    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.366999    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367002    2569 flags.go:64] FLAG: --cgroup-root=""
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367005    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367008    2569 flags.go:64] FLAG: --client-ca-file=""
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367010    2569 flags.go:64] FLAG: --cloud-config=""
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367013    2569 flags.go:64] FLAG: --cloud-provider="external"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367016    2569 flags.go:64] FLAG: --cluster-dns="[]"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367020    2569 flags.go:64] FLAG: --cluster-domain=""
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367022    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367025    2569 flags.go:64] FLAG: --config-dir=""
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367028    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367032    2569 flags.go:64] FLAG: --container-log-max-files="5"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367035    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367038    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367041    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367044    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367047    2569 flags.go:64] FLAG: --contention-profiling="false"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367050    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367053    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367056    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367059    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 11:17:12.370115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367063    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367066    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367069    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367072    2569 flags.go:64] FLAG: --enable-load-reader="false"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367075    2569 flags.go:64] FLAG: --enable-server="true"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367078    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367083    2569 flags.go:64] FLAG: --event-burst="100"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367086    2569 flags.go:64] FLAG: --event-qps="50"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367089    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367092    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367099    2569 flags.go:64] FLAG: --eviction-hard=""
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367103    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367106    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367109    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367112    2569 flags.go:64] FLAG: --eviction-soft=""
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367115    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367118    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367121    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367124    2569 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367127    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367130    2569 flags.go:64] FLAG: --fail-swap-on="true"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367133    2569 flags.go:64] FLAG: --feature-gates=""
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367136    2569 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367139    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367143    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 11:17:12.370718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367146    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367148    2569 flags.go:64] FLAG: --healthz-port="10248"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367152    2569 flags.go:64] FLAG: --help="false"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367154    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-141-158.ec2.internal"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367157    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367160    2569 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367163    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367167    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367170    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367173    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367176    2569 flags.go:64] FLAG: --image-service-endpoint=""
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367179    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367182    2569 flags.go:64] FLAG: --kube-api-burst="100"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367185    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367188    2569 flags.go:64] FLAG: --kube-api-qps="50"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367191    2569 flags.go:64] FLAG: --kube-reserved=""
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367194    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367198    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367201    2569 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367204    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367207    2569 flags.go:64] FLAG: --lock-file=""
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367210    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367213    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367216    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 11:17:12.371354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367222    2569 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367225    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367227    2569 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367230    2569 flags.go:64] FLAG: --logging-format="text"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367233    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367237    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367240    2569 flags.go:64] FLAG: --manifest-url=""
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367243    2569 flags.go:64] FLAG: --manifest-url-header=""
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367247    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367250    2569 flags.go:64] FLAG: --max-open-files="1000000"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367255    2569 flags.go:64] FLAG: --max-pods="110"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367258    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367261    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367264    2569 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367267    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367270    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367273    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367276    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367288    2569 flags.go:64] FLAG: --node-status-max-images="50"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367291    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367294    2569 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367297    2569 flags.go:64] FLAG: --pod-cidr=""
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367300    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 11:17:12.371959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367306    2569 flags.go:64] FLAG: --pod-manifest-path=""
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367309    2569 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367312    2569 flags.go:64] FLAG: --pods-per-core="0"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367316    2569 flags.go:64] FLAG: --port="10250"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367319    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367322    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0216a6b70cecd2ca7"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367325    2569 flags.go:64] FLAG: --qos-reserved=""
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367328    2569 flags.go:64] FLAG: --read-only-port="10255"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367331    2569 flags.go:64] FLAG: --register-node="true"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367334    2569 flags.go:64] FLAG: --register-schedulable="true"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367337    2569 flags.go:64] FLAG: --register-with-taints=""
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367340    2569 flags.go:64] FLAG: --registry-burst="10"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367343    2569 flags.go:64] FLAG: --registry-qps="5"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367346    2569 flags.go:64] FLAG: --reserved-cpus=""
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367348    2569 flags.go:64] FLAG: --reserved-memory=""
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367352    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367355    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367358    2569 flags.go:64] FLAG: --rotate-certificates="false"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367361    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367364    2569 flags.go:64] FLAG: --runonce="false"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367367    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367370    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367373    2569 flags.go:64] FLAG: --seccomp-default="false"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367376    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367378    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367381    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 11:17:12.372643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367384    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367387    2569 flags.go:64] FLAG: --storage-driver-password="root"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367390    2569 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367392    2569 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367395    2569 flags.go:64] FLAG: --storage-driver-user="root"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367398    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367401    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367407    2569 flags.go:64] FLAG: --system-cgroups=""
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367410    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367416    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367419    2569 flags.go:64] FLAG: --tls-cert-file=""
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367422    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367425    2569 flags.go:64] FLAG: --tls-min-version=""
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367428    2569 flags.go:64] FLAG: --tls-private-key-file=""
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367431    2569 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367434    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367436    2569 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367439    2569 flags.go:64] FLAG: --v="2"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367443    2569 flags.go:64] FLAG: --version="false"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367447    2569 flags.go:64] FLAG: --vmodule=""
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367451    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367454    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367549    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367553    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:12.373354 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367557    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367560    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367563    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367566    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367569    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367571    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367574    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367577    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367579    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367582    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367584    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367587    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367589    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367593    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367597    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367599    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367602    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367606    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367609    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367612    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:12.373958 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367615    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367618    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367620    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367623    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367626    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367628    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367631    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367633    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367636    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367638    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367641    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367643    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367648    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367650    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367653    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367656    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367658    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367661    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367663    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367666    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:12.374466 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367668    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367670    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367673    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367675    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367678    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367681    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367683    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367686    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367689    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367693    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367696    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367699    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367701    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367704    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367706    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367709    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367711    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367714    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367716    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367720    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:12.374977 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367722    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367725    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367728    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367730    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367734    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367737    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367739    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367742    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367744    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367747    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367749    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367752    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367754    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367756    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367759    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367761    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367764    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367784    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367788    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:12.375470 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367791    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367794    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367798    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367802    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.367804    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.367810    2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.375874    2569 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.375890    2569 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375934    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375939    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375943    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375946    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375949    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375951    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375954    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375957    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:12.375972 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375959    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375962    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375965    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375967    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375970    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375972    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375975    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375978    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375981    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375984    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375987    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375990    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.375992    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376000    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376003    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376006    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376008    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376012    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376015    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376017    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:12.376367 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376020    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376022    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376025    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376027    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376030    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376032    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376035    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376038    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376041    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376043    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376046    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376048    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376051    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376054    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376057    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376060    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376063    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376065    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376068    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376070    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:12.376911 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376073    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376076    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376079    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376081    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376084    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376086    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376088    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376091    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376093    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376096    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376100    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376104    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376107    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376110    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376112    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376116    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376120    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376123    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376126    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:12.377400 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376128    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376131    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376133    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376136    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376138    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376141    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376144    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376147    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376150    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376153    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376156    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376159    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376161    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376164    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376166    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376169    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376171    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376174    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:12.377883 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376176    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.376181    2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376275    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376280    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376283    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376286    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376289    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376291    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376294    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376296    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376299    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376303    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376307    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376310    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376313    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:12.378321 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376315    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376318    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376320    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376323    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376325    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376328    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376331    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376333    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376336    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376339    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376342    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376345    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376348    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376351    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376353    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376356    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376358    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376360    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376363    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:12.378691 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376365    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376368    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376370    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376373    2569 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376376    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376378    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376381    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376383    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376386    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376388    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376391    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376394    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376396    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376398    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376401    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376403    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376406    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376408    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376412    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:12.379180 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376414    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376418    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376421    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376423    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376427    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376429    2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376431    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376434    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376437    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376439    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376442    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376444    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376447    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376449    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376452    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376454    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376456    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376459    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376462    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376464    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:12.379660 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376466    2569 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376469    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376471    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376474    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376476    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376479    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376481    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376484    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376486    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376488    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376491    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376493    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376496    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376499    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:12.376502    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.376506    2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:12.380222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.377115    2569 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 11:17:12.380612 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.379061    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 11:17:12.380612 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.380039    2569 server.go:1019] "Starting client certificate rotation"
May 25 11:17:12.380612 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.380137    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:12.380612 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.380170    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:12.405642 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.405623    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:12.406858 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.406844    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:12.420897 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.420880    2569 log.go:25] "Validated CRI v1 runtime API"
May 25 11:17:12.426186 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.426169    2569 log.go:25] "Validated CRI v1 image API"
May 25 11:17:12.427443 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.427427    2569 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 11:17:12.430841 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.430820    2569 fs.go:135] Filesystem UUIDs: map[0ab1893a-a0f7-4651-8d77-e8f9028f31b2:/dev/nvme0n1p4 7592b44c-6c38-406c-b822-1569b2ce9f3a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
May 25 11:17:12.430915 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.430839    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 11:17:12.435044 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.434942    2569 manager.go:217] Machine: {Timestamp:2026-05-25 11:17:12.434413927 +0000 UTC m=+0.430588731 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101773 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22276f7d34f5413182923e3a230dd4 SystemUUID:ec22276f-7d34-f541-3182-923e3a230dd4 BootID:89eb2f25-18c1-4d59-8040-2838e8ad4bff Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6a:1e:20:4e:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6a:1e:20:4e:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:fa:2e:34:63:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 11:17:12.435895 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.435883    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 11:17:12.435977 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.435965    2569 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 11:17:12.436880 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.436857    2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:12.439382 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.439352    2569 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 11:17:12.439509 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.439384    2569 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-158.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 11:17:12.439557 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.439518    2569 topology_manager.go:138] "Creating topology manager with none policy"
May 25 11:17:12.439557 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.439526    2569 container_manager_linux.go:306] "Creating device plugin manager"
May 25 11:17:12.439557 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.439538    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:12.439557 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.439552    2569 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:12.441285 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.441274    2569 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:12.441533 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.441524    2569 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 11:17:12.445833 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.445820    2569 kubelet.go:491] "Attempting to sync node with API server"
May 25 11:17:12.445879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.445837    2569 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 11:17:12.445879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.445854    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 11:17:12.445879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.445864    2569 kubelet.go:397] "Adding apiserver pod source"
May 25 11:17:12.445879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.445872    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 11:17:12.447271 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.447258    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:12.447321 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.447279    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:12.450789 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.450760    2569 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 11:17:12.452516 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.452482    2569 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 11:17:12.453478 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453465    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 11:17:12.453523 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453482    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 11:17:12.453523 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453488    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 11:17:12.453523 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453493    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 11:17:12.453523 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453501    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 11:17:12.453523 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453507    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 11:17:12.453523 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453514    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 11:17:12.453523 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453519    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 11:17:12.453523 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453526    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 11:17:12.453728 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453533    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 11:17:12.453728 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453541    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 11:17:12.453804 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.453791    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 11:17:12.454672 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.454663    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 11:17:12.454710 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.454674    2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 11:17:12.458076 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.458044    2569 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-158.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 11:17:12.458187 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.458175    2569 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 11:17:12.458224 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.458208    2569 server.go:1295] "Started kubelet"
May 25 11:17:12.458325 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.458277    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 11:17:12.458967 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.458944    2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 11:17:12.458967 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.458947    2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 11:17:12.459109 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.458999    2569 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 11:17:12.459109 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.459048    2569 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 11:17:12.459742 ip-10-0-141-158 systemd[1]: Started Kubernetes Kubelet.
May 25 11:17:12.460960 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.460945    2569 server.go:317] "Adding debug handlers to kubelet server"
May 25 11:17:12.462208 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.462191    2569 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 11:17:12.465596 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.464444    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-158.ec2.internal.18b2cad825a69189  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-158.ec2.internal,UID:ip-10-0-141-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-158.ec2.internal,},FirstTimestamp:2026-05-25 11:17:12.458187145 +0000 UTC m=+0.454361949,LastTimestamp:2026-05-25 11:17:12.458187145 +0000 UTC m=+0.454361949,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-158.ec2.internal,}"
May 25 11:17:12.467207 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.467189    2569 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 11:17:12.468136 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.468119    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l7nf5"
May 25 11:17:12.468482 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.468457    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 11:17:12.469185 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.469170    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 11:17:12.469261 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.469248    2569 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 11:17:12.469261 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.469259    2569 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 11:17:12.469348 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.469310    2569 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 11:17:12.469394 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.469369    2569 reconstruct.go:97] "Volume reconstruction finished"
May 25 11:17:12.469394 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.469380    2569 reconciler.go:26] "Reconciler: start to sync state"
May 25 11:17:12.471372 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.471347    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:12.471881 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.471862    2569 factory.go:55] Registering systemd factory
May 25 11:17:12.471881 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.471881    2569 factory.go:223] Registration of the systemd container factory successfully
May 25 11:17:12.472251 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.472236    2569 factory.go:153] Registering CRI-O factory
May 25 11:17:12.472321 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.472257    2569 factory.go:223] Registration of the crio container factory successfully
May 25 11:17:12.472321 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.472315    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 11:17:12.472405 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.472336    2569 factory.go:103] Registering Raw factory
May 25 11:17:12.472405 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.472387    2569 manager.go:1196] Started watching for new ooms in manager
May 25 11:17:12.472927 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.472912    2569 manager.go:319] Starting recovery of all containers
May 25 11:17:12.474452 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.474424    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 11:17:12.475111 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.475078    2569 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 11:17:12.477734 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.477717    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l7nf5"
May 25 11:17:12.484165 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.484057    2569 manager.go:324] Recovery completed
May 25 11:17:12.488187 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.488177    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:12.490980 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.490965    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:12.491051 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.490991    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:12.491051 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.491005    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:12.491425 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.491410    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 11:17:12.491425 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.491422    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 11:17:12.491502 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.491436    2569 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:12.495130 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.495115    2569 policy_none.go:49] "None policy: Start"
May 25 11:17:12.495130 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.495131    2569 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 11:17:12.495233 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.495142    2569 state_mem.go:35] "Initializing new in-memory state store"
May 25 11:17:12.527415 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.527403    2569 manager.go:341] "Starting Device Plugin manager"
May 25 11:17:12.548311 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.527428    2569 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 11:17:12.548311 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.527438    2569 server.go:85] "Starting device plugin registration server"
May 25 11:17:12.548311 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.527638    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 11:17:12.548311 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.527649    2569 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 11:17:12.548311 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.527721    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 11:17:12.548311 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.527806    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 11:17:12.548311 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.527815    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 11:17:12.548311 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.528328    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 11:17:12.548311 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.528360    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:12.563810 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.563781    2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 11:17:12.564872 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.564852    2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 11:17:12.564945 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.564877    2569 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 11:17:12.564945 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.564899    2569 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 11:17:12.564945 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.564905    2569 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 11:17:12.564945 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.564935    2569 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 11:17:12.567325 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.567309    2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:12.628510 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.628450    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:12.629698 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.629683    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:12.629762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.629713    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:12.629762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.629723    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:12.629762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.629744    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-158.ec2.internal"
May 25 11:17:12.639928 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.639908    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-158.ec2.internal"
May 25 11:17:12.640005 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.639931    2569 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-158.ec2.internal\": node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:12.665532 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.665509    2569 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal"]
May 25 11:17:12.665616 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.665572    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:12.665693 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.665677    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:12.667650 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.667630    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:12.667718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.667660    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:12.667718 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.667673    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:12.669920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.669908    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:12.670085 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.670071    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.670165 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.670105    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:12.670618 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.670601    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:12.670709 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.670630    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:12.670709 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.670605    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:12.670709 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.670656    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:12.670709 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.670665    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:12.670709 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.670641    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:12.672841 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.672829    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.672890 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.672854    2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:12.673439 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.673424    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:12.673516 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.673454    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:12.673516 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.673467    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:12.705299 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.705281    2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-158.ec2.internal\" not found" node="ip-10-0-141-158.ec2.internal"
May 25 11:17:12.709547 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.709532    2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-158.ec2.internal\" not found" node="ip-10-0-141-158.ec2.internal"
May 25 11:17:12.765823 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.765797    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:12.771252 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.771236    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de79b3f5b34b5816fa18413a54ce6157-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal\" (UID: \"de79b3f5b34b5816fa18413a54ce6157\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.771311 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.771260    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de79b3f5b34b5816fa18413a54ce6157-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal\" (UID: \"de79b3f5b34b5816fa18413a54ce6157\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.771311 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.771279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c03ce34e78836af46c4b9aafeba94245-config\") pod \"kube-apiserver-proxy-ip-10-0-141-158.ec2.internal\" (UID: \"c03ce34e78836af46c4b9aafeba94245\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.866212 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.866186    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:12.872076 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.872059    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de79b3f5b34b5816fa18413a54ce6157-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal\" (UID: \"de79b3f5b34b5816fa18413a54ce6157\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.872141 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.872083    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de79b3f5b34b5816fa18413a54ce6157-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal\" (UID: \"de79b3f5b34b5816fa18413a54ce6157\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.872141 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.872100    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c03ce34e78836af46c4b9aafeba94245-config\") pod \"kube-apiserver-proxy-ip-10-0-141-158.ec2.internal\" (UID: \"c03ce34e78836af46c4b9aafeba94245\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.872141 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.872140    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c03ce34e78836af46c4b9aafeba94245-config\") pod \"kube-apiserver-proxy-ip-10-0-141-158.ec2.internal\" (UID: \"c03ce34e78836af46c4b9aafeba94245\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.872271 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.872158    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de79b3f5b34b5816fa18413a54ce6157-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal\" (UID: \"de79b3f5b34b5816fa18413a54ce6157\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.872271 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:12.872192    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de79b3f5b34b5816fa18413a54ce6157-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal\" (UID: \"de79b3f5b34b5816fa18413a54ce6157\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal"
May 25 11:17:12.966615 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:12.966550    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:13.008025 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.008005    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal"
May 25 11:17:13.012747 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.012728    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal"
May 25 11:17:13.067163 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:13.067134    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:13.167630 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:13.167608    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:13.268139 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:13.268074    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:13.339936 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.339916    2569 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:13.368311 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:13.368291    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:13.379698 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.379678    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 11:17:13.379850 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.379832    2569 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:13.379899 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.379850    2569 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:13.469249 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:13.469218    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:13.469249 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.469234    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 11:17:13.479115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.479097    2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:13.480137 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.480099    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 11:12:12 +0000 UTC" deadline="2028-01-05 11:37:03.179086434 +0000 UTC"
May 25 11:17:13.480137 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.480125    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14160h19m49.698964916s"
May 25 11:17:13.513334 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.513317    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-56tlm"
May 25 11:17:13.519452 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.519414    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-56tlm"
May 25 11:17:13.519524 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:13.519440    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde79b3f5b34b5816fa18413a54ce6157.slice/crio-7191e81d1c54b3df3cbc6908d16e54c5af292cc146ea74e6c6f3ae3f1f3e2183 WatchSource:0}: Error finding container 7191e81d1c54b3df3cbc6908d16e54c5af292cc146ea74e6c6f3ae3f1f3e2183: Status 404 returned error can't find the container with id 7191e81d1c54b3df3cbc6908d16e54c5af292cc146ea74e6c6f3ae3f1f3e2183
May 25 11:17:13.519727 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:13.519702    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03ce34e78836af46c4b9aafeba94245.slice/crio-66f325c14c99e448d928ebb4169238d9a4fe897ed280ff884a6f4bf1fc07b494 WatchSource:0}: Error finding container 66f325c14c99e448d928ebb4169238d9a4fe897ed280ff884a6f4bf1fc07b494: Status 404 returned error can't find the container with id 66f325c14c99e448d928ebb4169238d9a4fe897ed280ff884a6f4bf1fc07b494
May 25 11:17:13.524032 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.524018    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:17:13.568081 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.568029    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal" event={"ID":"de79b3f5b34b5816fa18413a54ce6157","Type":"ContainerStarted","Data":"7191e81d1c54b3df3cbc6908d16e54c5af292cc146ea74e6c6f3ae3f1f3e2183"}
May 25 11:17:13.568987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.568964    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal" event={"ID":"c03ce34e78836af46c4b9aafeba94245","Type":"ContainerStarted","Data":"66f325c14c99e448d928ebb4169238d9a4fe897ed280ff884a6f4bf1fc07b494"}
May 25 11:17:13.570139 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:13.570124    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:13.579038 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.579019    2569 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:13.670201 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:13.670174    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:13.770697 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:13.770631    2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-158.ec2.internal\" not found"
May 25 11:17:13.841190 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.841166    2569 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:13.870392 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.870365    2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal"
May 25 11:17:13.882591 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.882569    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:13.883745 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.883722    2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal"
May 25 11:17:13.893915 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:13.893893    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:14.447584 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.447546    2569 apiserver.go:52] "Watching apiserver"
May 25 11:17:14.455056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.455037    2569 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 11:17:14.455478 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.455452    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-nwq6c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal","openshift-multus/multus-jfr78","openshift-multus/network-metrics-daemon-z2l9j","kube-system/konnectivity-agent-hjknq","openshift-multus/multus-additional-cni-plugins-b5mdm","openshift-network-diagnostics/network-check-target-gzg8z","openshift-network-operator/iptables-alerter-5tbtf","openshift-ovn-kubernetes/ovnkube-node-jbtgj","kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx","openshift-cluster-node-tuning-operator/tuned-vlsbt","openshift-dns/node-resolver-sg6vv"]
May 25 11:17:14.457946 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.457925    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:14.458037 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:14.458013    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:14.460097 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.460078    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jfr78"
May 25 11:17:14.462840 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.462630    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 11:17:14.462840 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.462672    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 11:17:14.462840 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.462637    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 11:17:14.462840 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.462675    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 11:17:14.462840 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.462800    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpp88\""
May 25 11:17:14.463074 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.463022    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:14.463107 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:14.463080    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:14.465259 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.465244    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:14.468383 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.467910    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 11:17:14.468383 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.467912    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lkmss\""
May 25 11:17:14.468383 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.468237    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 11:17:14.468572 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.468541    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.470994 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.470975    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 11:17:14.471411 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.471188    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 11:17:14.471411 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.471274    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mb4qd\""
May 25 11:17:14.471411 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.471339    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.473463 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.473442    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 11:17:14.473602 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.473579    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 11:17:14.473712 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.473664    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 11:17:14.473840 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.473819    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-plq52\""
May 25 11:17:14.475902 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.475882    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.476164 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.476145    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.478369 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.478349    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.480310 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480289    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-cnibin\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.480417 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b32b422f-cd6e-4bcb-a52a-9c7b3e895676-host\") pod \"node-ca-nwq6c\" (UID: \"b32b422f-cd6e-4bcb-a52a-9c7b3e895676\") " pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.480417 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-run-netns\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480417 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480369    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-conf-dir\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480417 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480410    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnlr9\" (UniqueName: \"kubernetes.io/projected/c8d62b8b-fa29-494d-a37a-3436347370ed-kube-api-access-wnlr9\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:14.480622 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480438    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.480622 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480464    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-var-lib-cni-bin\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480622 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480487    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-var-lib-cni-multus\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480622 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480510    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-os-release\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.480622 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480541    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.480622 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b32b422f-cd6e-4bcb-a52a-9c7b3e895676-serviceca\") pod \"node-ca-nwq6c\" (UID: \"b32b422f-cd6e-4bcb-a52a-9c7b3e895676\") " pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.480622 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-system-cni-dir\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480654    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480688    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-cni-dir\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-run-multus-certs\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480733    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3126fb3-2cba-43c5-9244-4cc915eac03d-konnectivity-ca\") pod \"konnectivity-agent-hjknq\" (UID: \"b3126fb3-2cba-43c5-9244-4cc915eac03d\") " pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480739    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480756    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-socket-dir-parent\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-hostroot\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480860    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480894    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhtr\" (UniqueName: \"kubernetes.io/projected/b32b422f-cd6e-4bcb-a52a-9c7b3e895676-kube-api-access-jhhtr\") pod \"node-ca-nwq6c\" (UID: \"b32b422f-cd6e-4bcb-a52a-9c7b3e895676\") " pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-system-cni-dir\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480925    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-run-k8s-cni-cncf-io\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480963    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9wnd\" (UniqueName: \"kubernetes.io/projected/295056bc-0e37-4116-bbd8-00f40557f5d5-kube-api-access-l9wnd\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.480987 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.480985    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3126fb3-2cba-43c5-9244-4cc915eac03d-agent-certs\") pod \"konnectivity-agent-hjknq\" (UID: \"b3126fb3-2cba-43c5-9244-4cc915eac03d\") " pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481008    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-cnibin\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481060    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-os-release\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481091    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/295056bc-0e37-4116-bbd8-00f40557f5d5-cni-binary-copy\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-var-lib-kubelet\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481177    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-daemon-config\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481232    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481267    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481266    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-cni-binary-copy\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481311    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52xq5\" (UniqueName: \"kubernetes.io/projected/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-kube-api-access-52xq5\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.481597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.481338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-etc-kubernetes\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.482366 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482348    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 11:17:14.482366 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482358    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 11:17:14.482482 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482367    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 11:17:14.482482 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482407    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 11:17:14.482698 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482673    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 11:17:14.482698 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482699    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:14.482869 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482675    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 11:17:14.482869 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482675    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qlvgr\""
May 25 11:17:14.482970 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482879    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4vbk9\""
May 25 11:17:14.482970 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.482960    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.483575 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.483558    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 11:17:14.483575 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.483571    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bql8n\""
May 25 11:17:14.483742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.483727    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 11:17:14.483928 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.483911    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4f9nv\""
May 25 11:17:14.484017 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.484001    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 11:17:14.484073 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.484017    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:14.484128 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.484087    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 11:17:14.486356 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.486338    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 11:17:14.486445 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.486423    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 11:17:14.486687 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.486673    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bwcnv\""
May 25 11:17:14.520380 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.520357    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:13 +0000 UTC" deadline="2028-01-26 21:16:41.37128207 +0000 UTC"
May 25 11:17:14.520380 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.520379    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14673h59m26.850905648s"
May 25 11:17:14.570287 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.570262    2569 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 11:17:14.582362 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582334    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-var-lib-cni-bin\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.582497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582375    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-var-lib-cni-multus\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.582497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582400    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-var-lib-cni-bin\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.582497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582450    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-var-lib-cni-multus\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.582497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582405    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.582497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-socket-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582524    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582537    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-cni-bin\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582566    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-modprobe-d\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582592    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa1e4cb-c587-401c-8dc7-a6bc27de70d3-tmp-dir\") pod \"node-resolver-sg6vv\" (UID: \"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3\") " pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28e6f486-3b90-4ab3-8088-d41e46546740-iptables-alerter-script\") pod \"iptables-alerter-5tbtf\" (UID: \"28e6f486-3b90-4ab3-8088-d41e46546740\") " pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582655    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-node-log\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582682    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f67dfd38-152e-40bf-beb0-430505aa58fe-ovn-node-metrics-cert\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582705    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr55r\" (UniqueName: \"kubernetes.io/projected/f67dfd38-152e-40bf-beb0-430505aa58fe-kube-api-access-rr55r\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.582736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582729    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-hostroot\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582751    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582791    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhtr\" (UniqueName: \"kubernetes.io/projected/b32b422f-cd6e-4bcb-a52a-9c7b3e895676-kube-api-access-jhhtr\") pod \"node-ca-nwq6c\" (UID: \"b32b422f-cd6e-4bcb-a52a-9c7b3e895676\") " pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-run\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582818    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-hostroot\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bgcd\" (UniqueName: \"kubernetes.io/projected/28e6f486-3b90-4ab3-8088-d41e46546740-kube-api-access-4bgcd\") pod \"iptables-alerter-5tbtf\" (UID: \"28e6f486-3b90-4ab3-8088-d41e46546740\") " pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-etc-selinux\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582886    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-system-cni-dir\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582944    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-system-cni-dir\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582964    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l9wnd\" (UniqueName: \"kubernetes.io/projected/295056bc-0e37-4116-bbd8-00f40557f5d5-kube-api-access-l9wnd\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.582995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-tmp\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-systemd-units\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583114    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/295056bc-0e37-4116-bbd8-00f40557f5d5-cni-binary-copy\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583142    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583168    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-cni-binary-copy\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583194    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52xq5\" (UniqueName: \"kubernetes.io/projected/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-kube-api-access-52xq5\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.583221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583223    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-systemd\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583248    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-tuned\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:14.583270    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8nj\" (UniqueName: \"kubernetes.io/projected/fdba9b03-f691-4b5e-840c-2bfb07c72094-kube-api-access-qx8nj\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583301    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-etc-kubernetes\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583326    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-cnibin\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-run-netns\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f67dfd38-152e-40bf-beb0-430505aa58fe-ovnkube-script-lib\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583429    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-lib-modules\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fa1e4cb-c587-401c-8dc7-a6bc27de70d3-hosts-file\") pod \"node-resolver-sg6vv\" (UID: \"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3\") " pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583481    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbgc8\" (UniqueName: \"kubernetes.io/projected/9fa1e4cb-c587-401c-8dc7-a6bc27de70d3-kube-api-access-fbgc8\") pod \"node-resolver-sg6vv\" (UID: \"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3\") " pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583490    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-etc-kubernetes\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583505    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-device-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583531    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-run-netns\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583568    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-cnibin\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583576    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-run-netns\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583448    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.584016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583596    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wnlr9\" (UniqueName: \"kubernetes.io/projected/c8d62b8b-fa29-494d-a37a-3436347370ed-kube-api-access-wnlr9\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583628    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583656    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-sysctl-conf\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28e6f486-3b90-4ab3-8088-d41e46546740-host-slash\") pod \"iptables-alerter-5tbtf\" (UID: \"28e6f486-3b90-4ab3-8088-d41e46546740\") " pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583709    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-cni-binary-copy\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:14.583722    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs podName:c8d62b8b-fa29-494d-a37a-3436347370ed nodeName:}" failed. No retries permitted until 2026-05-25 11:17:15.083678983 +0000 UTC m=+3.079853790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs") pod "network-metrics-daemon-z2l9j" (UID: "c8d62b8b-fa29-494d-a37a-3436347370ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583710    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/295056bc-0e37-4116-bbd8-00f40557f5d5-cni-binary-copy\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583756    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-os-release\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583799    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b32b422f-cd6e-4bcb-a52a-9c7b3e895676-serviceca\") pod \"node-ca-nwq6c\" (UID: \"b32b422f-cd6e-4bcb-a52a-9c7b3e895676\") " pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583833    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-os-release\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583844    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f67dfd38-152e-40bf-beb0-430505aa58fe-ovnkube-config\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583920    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-system-cni-dir\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-slash\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583982    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-system-cni-dir\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.583997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-log-socket\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.584845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584040    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-cni-netd\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-registration-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584129    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-cni-dir\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584200    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-cni-dir\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584200    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b32b422f-cd6e-4bcb-a52a-9c7b3e895676-serviceca\") pod \"node-ca-nwq6c\" (UID: \"b32b422f-cd6e-4bcb-a52a-9c7b3e895676\") " pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584241    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-run-multus-certs\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584270    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3126fb3-2cba-43c5-9244-4cc915eac03d-konnectivity-ca\") pod \"konnectivity-agent-hjknq\" (UID: \"b3126fb3-2cba-43c5-9244-4cc915eac03d\") " pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584295    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-kubelet\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-etc-openvswitch\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584322    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-run-multus-certs\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584344    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-host\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-sys-fs\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584399    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-socket-dir-parent\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584426    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-run-systemd\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584482    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-run-k8s-cni-cncf-io\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584492    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-socket-dir-parent\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.585504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584506    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3126fb3-2cba-43c5-9244-4cc915eac03d-agent-certs\") pod \"konnectivity-agent-hjknq\" (UID: \"b3126fb3-2cba-43c5-9244-4cc915eac03d\") " pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-run-openvswitch\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584544    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-run-k8s-cni-cncf-io\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-sysconfig\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584581    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b32b422f-cd6e-4bcb-a52a-9c7b3e895676-host\") pod \"node-ca-nwq6c\" (UID: \"b32b422f-cd6e-4bcb-a52a-9c7b3e895676\") " pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584604    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-cnibin\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584627    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-os-release\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584652    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-var-lib-kubelet\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584698    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584729    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-daemon-config\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584756    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-sysctl-d\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584793    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3126fb3-2cba-43c5-9244-4cc915eac03d-konnectivity-ca\") pod \"konnectivity-agent-hjknq\" (UID: \"b3126fb3-2cba-43c5-9244-4cc915eac03d\") " pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584800    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-sys\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-var-lib-kubelet\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584848    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-var-lib-openvswitch\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584869    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-host-var-lib-kubelet\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584900    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f67dfd38-152e-40bf-beb0-430505aa58fe-env-overrides\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.586084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-kubernetes\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.586605 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584959    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-cnibin\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586605 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584959    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-conf-dir\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586605 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584989    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-os-release\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586605 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-run-ovn\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.586605 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.584996    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-conf-dir\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.586605 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.585022    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b32b422f-cd6e-4bcb-a52a-9c7b3e895676-host\") pod \"node-ca-nwq6c\" (UID: \"b32b422f-cd6e-4bcb-a52a-9c7b3e895676\") " pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.586605 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.585052    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpgz\" (UniqueName: \"kubernetes.io/projected/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-kube-api-access-9bpgz\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.586605 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.585083    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.586605 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.585240    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/295056bc-0e37-4116-bbd8-00f40557f5d5-multus-daemon-config\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.588287 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.588264    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3126fb3-2cba-43c5-9244-4cc915eac03d-agent-certs\") pod \"konnectivity-agent-hjknq\" (UID: \"b3126fb3-2cba-43c5-9244-4cc915eac03d\") " pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:14.591879 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:14.591860    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:14.591879 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:14.591881    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:14.592051 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:14.591891    2569 projected.go:194] Error preparing data for projected volume kube-api-access-9nw7d for pod openshift-network-diagnostics/network-check-target-gzg8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:14.592051 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:14.591936    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d podName:607c3221-179d-44be-81d0-930ca62118ad nodeName:}" failed. No retries permitted until 2026-05-25 11:17:15.091921938 +0000 UTC m=+3.088096754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9nw7d" (UniqueName: "kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d") pod "network-check-target-gzg8z" (UID: "607c3221-179d-44be-81d0-930ca62118ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:14.594216 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.594196    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhtr\" (UniqueName: \"kubernetes.io/projected/b32b422f-cd6e-4bcb-a52a-9c7b3e895676-kube-api-access-jhhtr\") pod \"node-ca-nwq6c\" (UID: \"b32b422f-cd6e-4bcb-a52a-9c7b3e895676\") " pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.594736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.594716    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52xq5\" (UniqueName: \"kubernetes.io/projected/9b5d0970-d81c-4ffb-9e1d-d2be86769fb9-kube-api-access-52xq5\") pod \"multus-additional-cni-plugins-b5mdm\" (UID: \"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9\") " pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.595156 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.595137    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9wnd\" (UniqueName: \"kubernetes.io/projected/295056bc-0e37-4116-bbd8-00f40557f5d5-kube-api-access-l9wnd\") pod \"multus-jfr78\" (UID: \"295056bc-0e37-4116-bbd8-00f40557f5d5\") " pod="openshift-multus/multus-jfr78"
May 25 11:17:14.595237 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.595217    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnlr9\" (UniqueName: \"kubernetes.io/projected/c8d62b8b-fa29-494d-a37a-3436347370ed-kube-api-access-wnlr9\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:14.617379 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.617360    2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:14.685922 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.685895    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-modprobe-d\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.685922 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.685927    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa1e4cb-c587-401c-8dc7-a6bc27de70d3-tmp-dir\") pod \"node-resolver-sg6vv\" (UID: \"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3\") " pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.686145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.685950    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28e6f486-3b90-4ab3-8088-d41e46546740-iptables-alerter-script\") pod \"iptables-alerter-5tbtf\" (UID: \"28e6f486-3b90-4ab3-8088-d41e46546740\") " pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.686145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.685974    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-node-log\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.686145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686017    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f67dfd38-152e-40bf-beb0-430505aa58fe-ovn-node-metrics-cert\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.686145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686044    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-node-log\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.686145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686053    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rr55r\" (UniqueName: \"kubernetes.io/projected/f67dfd38-152e-40bf-beb0-430505aa58fe-kube-api-access-rr55r\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.686145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686063    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-modprobe-d\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.686145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686094    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-run\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.686145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686121    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4bgcd\" (UniqueName: \"kubernetes.io/projected/28e6f486-3b90-4ab3-8088-d41e46546740-kube-api-access-4bgcd\") pod \"iptables-alerter-5tbtf\" (UID: \"28e6f486-3b90-4ab3-8088-d41e46546740\") " pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.686145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686149    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-etc-selinux\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686175    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-tmp\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686188    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-run\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686200    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-systemd-units\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686240    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-systemd\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686267    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-tuned\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686282    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-etc-selinux\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686284    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa1e4cb-c587-401c-8dc7-a6bc27de70d3-tmp-dir\") pod \"node-resolver-sg6vv\" (UID: \"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3\") " pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686292    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8nj\" (UniqueName: \"kubernetes.io/projected/fdba9b03-f691-4b5e-840c-2bfb07c72094-kube-api-access-qx8nj\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686325    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-run-netns\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686333    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-systemd-units\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686351    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f67dfd38-152e-40bf-beb0-430505aa58fe-ovnkube-script-lib\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686376    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-lib-modules\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686413    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-systemd\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686425    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fa1e4cb-c587-401c-8dc7-a6bc27de70d3-hosts-file\") pod \"node-resolver-sg6vv\" (UID: \"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3\") " pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686452    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fbgc8\" (UniqueName: \"kubernetes.io/projected/9fa1e4cb-c587-401c-8dc7-a6bc27de70d3-kube-api-access-fbgc8\") pod \"node-resolver-sg6vv\" (UID: \"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3\") " pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686479    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-device-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.686546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686507    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28e6f486-3b90-4ab3-8088-d41e46546740-iptables-alerter-script\") pod \"iptables-alerter-5tbtf\" (UID: \"28e6f486-3b90-4ab3-8088-d41e46546740\") " pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686510    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-sysctl-conf\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686568    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28e6f486-3b90-4ab3-8088-d41e46546740-host-slash\") pod \"iptables-alerter-5tbtf\" (UID: \"28e6f486-3b90-4ab3-8088-d41e46546740\") " pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686599    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686625    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f67dfd38-152e-40bf-beb0-430505aa58fe-ovnkube-config\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686639    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-sysctl-conf\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686653    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-slash\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686678    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-lib-modules\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686687    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-log-socket\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686715    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fa1e4cb-c587-401c-8dc7-a6bc27de70d3-hosts-file\") pod \"node-resolver-sg6vv\" (UID: \"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3\") " pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686726    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-cni-netd\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686757    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-registration-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686761    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686800    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-device-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686821    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-kubelet\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686840    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-slash\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686878    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-kubelet\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.687327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686880    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28e6f486-3b90-4ab3-8088-d41e46546740-host-slash\") pod \"iptables-alerter-5tbtf\" (UID: \"28e6f486-3b90-4ab3-8088-d41e46546740\") " pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686729    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-log-socket\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.686931    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-cni-netd\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687113    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-run-netns\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687216    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-registration-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687262    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-etc-openvswitch\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687290    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-host\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687316    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-sys-fs\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687325    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f67dfd38-152e-40bf-beb0-430505aa58fe-ovnkube-config\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687343    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-run-systemd\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687373    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687389    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-host\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687406    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-run-openvswitch\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687431    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-sysconfig\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687460    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-run-systemd\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687461    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-sysctl-d\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687497    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-sys\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687524    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-var-lib-kubelet\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687549    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-var-lib-openvswitch\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687572    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-sysctl-d\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687577    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f67dfd38-152e-40bf-beb0-430505aa58fe-env-overrides\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687607    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-kubernetes\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687616    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f67dfd38-152e-40bf-beb0-430505aa58fe-ovnkube-script-lib\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687632    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-run-ovn\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687682    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-run-ovn\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687685    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-sys\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687716    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpgz\" (UniqueName: \"kubernetes.io/projected/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-kube-api-access-9bpgz\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687729    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687745    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687786    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-run-openvswitch\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687794    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-socket-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687835    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-cni-bin\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687837    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-sysconfig\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687870    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-kubernetes\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.688809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687429    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-etc-openvswitch\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.689387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687889    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-var-lib-kubelet\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.689387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687909    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-var-lib-openvswitch\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.689387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687927    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-sys-fs\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.689387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687943    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.689387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.687971    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f67dfd38-152e-40bf-beb0-430505aa58fe-host-cni-bin\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.689387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.688158    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdba9b03-f691-4b5e-840c-2bfb07c72094-socket-dir\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.689387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.688156    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f67dfd38-152e-40bf-beb0-430505aa58fe-env-overrides\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.689387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.689365    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-etc-tuned\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.689387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.689387    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-tmp\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.689749 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.689727    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f67dfd38-152e-40bf-beb0-430505aa58fe-ovn-node-metrics-cert\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.696050 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.696000    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8nj\" (UniqueName: \"kubernetes.io/projected/fdba9b03-f691-4b5e-840c-2bfb07c72094-kube-api-access-qx8nj\") pod \"aws-ebs-csi-driver-node-dhnhx\" (UID: \"fdba9b03-f691-4b5e-840c-2bfb07c72094\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.696050 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.696034    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpgz\" (UniqueName: \"kubernetes.io/projected/9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605-kube-api-access-9bpgz\") pod \"tuned-vlsbt\" (UID: \"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605\") " pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.696418 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.696392    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbgc8\" (UniqueName: \"kubernetes.io/projected/9fa1e4cb-c587-401c-8dc7-a6bc27de70d3-kube-api-access-fbgc8\") pod \"node-resolver-sg6vv\" (UID: \"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3\") " pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:14.696746 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.696723    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr55r\" (UniqueName: \"kubernetes.io/projected/f67dfd38-152e-40bf-beb0-430505aa58fe-kube-api-access-rr55r\") pod \"ovnkube-node-jbtgj\" (UID: \"f67dfd38-152e-40bf-beb0-430505aa58fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.697085 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.697067    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bgcd\" (UniqueName: \"kubernetes.io/projected/28e6f486-3b90-4ab3-8088-d41e46546740-kube-api-access-4bgcd\") pod \"iptables-alerter-5tbtf\" (UID: \"28e6f486-3b90-4ab3-8088-d41e46546740\") " pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.772130 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.772047    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jfr78"
May 25 11:17:14.778890 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.778873    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:14.788372 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.788353    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b5mdm"
May 25 11:17:14.793904 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.793888    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwq6c"
May 25 11:17:14.800392 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.800374    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5tbtf"
May 25 11:17:14.805938 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.805921    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:14.811444 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.811427    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx"
May 25 11:17:14.817024 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.817003    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vlsbt"
May 25 11:17:14.821538 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:14.821522    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sg6vv"
May 25 11:17:15.051734 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:15.051705    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab88c23_ea8f_4dc4_8e7a_bc7fe560a605.slice/crio-2818d10438b893c812f5b542e941de01fdc6d24cb66f8d3d421ae136939c2807 WatchSource:0}: Error finding container 2818d10438b893c812f5b542e941de01fdc6d24cb66f8d3d421ae136939c2807: Status 404 returned error can't find the container with id 2818d10438b893c812f5b542e941de01fdc6d24cb66f8d3d421ae136939c2807
May 25 11:17:15.053246 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:15.053227    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3126fb3_2cba_43c5_9244_4cc915eac03d.slice/crio-58626bfcc09a698ccb66bd79ccd2a139e0caf6285c5eaba5fb2ba0e8e7d6e5c0 WatchSource:0}: Error finding container 58626bfcc09a698ccb66bd79ccd2a139e0caf6285c5eaba5fb2ba0e8e7d6e5c0: Status 404 returned error can't find the container with id 58626bfcc09a698ccb66bd79ccd2a139e0caf6285c5eaba5fb2ba0e8e7d6e5c0
May 25 11:17:15.056482 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:15.056462    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdba9b03_f691_4b5e_840c_2bfb07c72094.slice/crio-0f77f365ae5a3196094d076057c8224fc36842501b4db92a125b637f65361f09 WatchSource:0}: Error finding container 0f77f365ae5a3196094d076057c8224fc36842501b4db92a125b637f65361f09: Status 404 returned error can't find the container with id 0f77f365ae5a3196094d076057c8224fc36842501b4db92a125b637f65361f09
May 25 11:17:15.057675 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:15.057541    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295056bc_0e37_4116_bbd8_00f40557f5d5.slice/crio-9b97ddea29028b29dbf4fc17326db59051fadcd4bc9c7d505630bd990d23183c WatchSource:0}: Error finding container 9b97ddea29028b29dbf4fc17326db59051fadcd4bc9c7d505630bd990d23183c: Status 404 returned error can't find the container with id 9b97ddea29028b29dbf4fc17326db59051fadcd4bc9c7d505630bd990d23183c
May 25 11:17:15.058404 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:15.058208    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa1e4cb_c587_401c_8dc7_a6bc27de70d3.slice/crio-d368e835271be37def5516b0b699e604b0c840a1f566ba3577ebf4b179cdcb23 WatchSource:0}: Error finding container d368e835271be37def5516b0b699e604b0c840a1f566ba3577ebf4b179cdcb23: Status 404 returned error can't find the container with id d368e835271be37def5516b0b699e604b0c840a1f566ba3577ebf4b179cdcb23
May 25 11:17:15.059046 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:15.058986    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67dfd38_152e_40bf_beb0_430505aa58fe.slice/crio-5c2bda951d74c7fbb9dbba55b0a31157afe1f579d4cc3012c34b23a5a3e46b87 WatchSource:0}: Error finding container 5c2bda951d74c7fbb9dbba55b0a31157afe1f579d4cc3012c34b23a5a3e46b87: Status 404 returned error can't find the container with id 5c2bda951d74c7fbb9dbba55b0a31157afe1f579d4cc3012c34b23a5a3e46b87
May 25 11:17:15.059959 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:15.059879    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5d0970_d81c_4ffb_9e1d_d2be86769fb9.slice/crio-af62d70cdbdea88126ed61e4f9a8d66e38e3b8a51fc84295046d0738a774e63d WatchSource:0}: Error finding container af62d70cdbdea88126ed61e4f9a8d66e38e3b8a51fc84295046d0738a774e63d: Status 404 returned error can't find the container with id af62d70cdbdea88126ed61e4f9a8d66e38e3b8a51fc84295046d0738a774e63d
May 25 11:17:15.063543 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:15.063519    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32b422f_cd6e_4bcb_a52a_9c7b3e895676.slice/crio-1e13774a569d00cc69df9fdb3fcec5379d88ed81cc3ed184596027bce807f49e WatchSource:0}: Error finding container 1e13774a569d00cc69df9fdb3fcec5379d88ed81cc3ed184596027bce807f49e: Status 404 returned error can't find the container with id 1e13774a569d00cc69df9fdb3fcec5379d88ed81cc3ed184596027bce807f49e
May 25 11:17:15.090930 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.090758    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:15.091022 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:15.090870    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:15.091058 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:15.091042    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs podName:c8d62b8b-fa29-494d-a37a-3436347370ed nodeName:}" failed. No retries permitted until 2026-05-25 11:17:16.091026869 +0000 UTC m=+4.087201659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs") pod "network-metrics-daemon-z2l9j" (UID: "c8d62b8b-fa29-494d-a37a-3436347370ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:15.191562 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.191537    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:15.191729 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:15.191710    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:15.191807 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:15.191737    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:15.191807 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:15.191752    2569 projected.go:194] Error preparing data for projected volume kube-api-access-9nw7d for pod openshift-network-diagnostics/network-check-target-gzg8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:15.191885 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:15.191829    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d podName:607c3221-179d-44be-81d0-930ca62118ad nodeName:}" failed. No retries permitted until 2026-05-25 11:17:16.191810024 +0000 UTC m=+4.187984818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nw7d" (UniqueName: "kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d") pod "network-check-target-gzg8z" (UID: "607c3221-179d-44be-81d0-930ca62118ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:15.520575 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.520533    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:13 +0000 UTC" deadline="2027-11-08 12:35:38.690398998 +0000 UTC"
May 25 11:17:15.520575 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.520573    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12769h18m23.169829622s"
May 25 11:17:15.566138 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.566066    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:15.566287 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:15.566198    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:15.579258 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.579203    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwq6c" event={"ID":"b32b422f-cd6e-4bcb-a52a-9c7b3e895676","Type":"ContainerStarted","Data":"1e13774a569d00cc69df9fdb3fcec5379d88ed81cc3ed184596027bce807f49e"}
May 25 11:17:15.599073 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.599010    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5mdm" event={"ID":"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9","Type":"ContainerStarted","Data":"af62d70cdbdea88126ed61e4f9a8d66e38e3b8a51fc84295046d0738a774e63d"}
May 25 11:17:15.600450 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.600398    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sg6vv" event={"ID":"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3","Type":"ContainerStarted","Data":"d368e835271be37def5516b0b699e604b0c840a1f566ba3577ebf4b179cdcb23"}
May 25 11:17:15.607222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.607174    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jfr78" event={"ID":"295056bc-0e37-4116-bbd8-00f40557f5d5","Type":"ContainerStarted","Data":"9b97ddea29028b29dbf4fc17326db59051fadcd4bc9c7d505630bd990d23183c"}
May 25 11:17:15.613451 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.613402    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx" event={"ID":"fdba9b03-f691-4b5e-840c-2bfb07c72094","Type":"ContainerStarted","Data":"0f77f365ae5a3196094d076057c8224fc36842501b4db92a125b637f65361f09"}
May 25 11:17:15.616670 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.616611    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hjknq" event={"ID":"b3126fb3-2cba-43c5-9244-4cc915eac03d","Type":"ContainerStarted","Data":"58626bfcc09a698ccb66bd79ccd2a139e0caf6285c5eaba5fb2ba0e8e7d6e5c0"}
May 25 11:17:15.634125 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.634038    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vlsbt" event={"ID":"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605","Type":"ContainerStarted","Data":"2818d10438b893c812f5b542e941de01fdc6d24cb66f8d3d421ae136939c2807"}
May 25 11:17:15.657792 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.657611    2569 generic.go:358] "Generic (PLEG): container finished" podID="de79b3f5b34b5816fa18413a54ce6157" containerID="de27b4a35232f6e57238240fb25d5c6f892de6f26a070900c196e287b8f76180" exitCode=0
May 25 11:17:15.657971 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.657926    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal" event={"ID":"de79b3f5b34b5816fa18413a54ce6157","Type":"ContainerDied","Data":"de27b4a35232f6e57238240fb25d5c6f892de6f26a070900c196e287b8f76180"}
May 25 11:17:15.660847 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.660823    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal" event={"ID":"c03ce34e78836af46c4b9aafeba94245","Type":"ContainerStarted","Data":"624eb5780ae0d7976114faeaa6eac6a01698fac2f8f86e99f9fef7596a7c6e8f"}
May 25 11:17:15.675314 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.675288    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5tbtf" event={"ID":"28e6f486-3b90-4ab3-8088-d41e46546740","Type":"ContainerStarted","Data":"439e12cccc5c7193087a5e5525ab6a042b494631d59230eb770f64aa1491c622"}
May 25 11:17:15.683208 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.683185    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerStarted","Data":"5c2bda951d74c7fbb9dbba55b0a31157afe1f579d4cc3012c34b23a5a3e46b87"}
May 25 11:17:15.770730 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:15.770385    2569 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:16.102520 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:16.102488    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:16.102695 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:16.102665    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:16.102844 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:16.102726    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs podName:c8d62b8b-fa29-494d-a37a-3436347370ed nodeName:}" failed. No retries permitted until 2026-05-25 11:17:18.102708105 +0000 UTC m=+6.098882910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs") pod "network-metrics-daemon-z2l9j" (UID: "c8d62b8b-fa29-494d-a37a-3436347370ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:16.202989 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:16.202910    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:16.203135 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:16.203097    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:16.203135 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:16.203114    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:16.203135 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:16.203127    2569 projected.go:194] Error preparing data for projected volume kube-api-access-9nw7d for pod openshift-network-diagnostics/network-check-target-gzg8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:16.203366 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:16.203181    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d podName:607c3221-179d-44be-81d0-930ca62118ad nodeName:}" failed. No retries permitted until 2026-05-25 11:17:18.203163604 +0000 UTC m=+6.199338416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nw7d" (UniqueName: "kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d") pod "network-check-target-gzg8z" (UID: "607c3221-179d-44be-81d0-930ca62118ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:16.568526 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:16.568014    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:16.568526 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:16.568140    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:16.711682 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:16.711617    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal" event={"ID":"de79b3f5b34b5816fa18413a54ce6157","Type":"ContainerStarted","Data":"c524c60c94da31cf37cdfb91a97a5de02801cf33171a55d872a4152e4e3e9ffd"}
May 25 11:17:16.725517 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:16.724255    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-158.ec2.internal" podStartSLOduration=3.724236769 podStartE2EDuration="3.724236769s" podCreationTimestamp="2026-05-25 11:17:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:15.685452534 +0000 UTC m=+3.681627347" watchObservedRunningTime="2026-05-25 11:17:16.724236769 +0000 UTC m=+4.720411582"
May 25 11:17:17.565466 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:17.565370    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:17.565649 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:17.565505    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:18.117505 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:18.117461    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:18.118061 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:18.117651    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:18.118061 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:18.117719    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs podName:c8d62b8b-fa29-494d-a37a-3436347370ed nodeName:}" failed. No retries permitted until 2026-05-25 11:17:22.117699883 +0000 UTC m=+10.113874679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs") pod "network-metrics-daemon-z2l9j" (UID: "c8d62b8b-fa29-494d-a37a-3436347370ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:18.220012 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:18.218106    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:18.220012 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:18.218287    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:18.220012 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:18.218305    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:18.220012 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:18.218326    2569 projected.go:194] Error preparing data for projected volume kube-api-access-9nw7d for pod openshift-network-diagnostics/network-check-target-gzg8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:18.220012 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:18.218386    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d podName:607c3221-179d-44be-81d0-930ca62118ad nodeName:}" failed. No retries permitted until 2026-05-25 11:17:22.218367458 +0000 UTC m=+10.214542266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nw7d" (UniqueName: "kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d") pod "network-check-target-gzg8z" (UID: "607c3221-179d-44be-81d0-930ca62118ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:18.566663 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:18.566141    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:18.566663 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:18.566268    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:19.439066 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.439008    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-158.ec2.internal" podStartSLOduration=6.438987358 podStartE2EDuration="6.438987358s" podCreationTimestamp="2026-05-25 11:17:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:16.72549601 +0000 UTC m=+4.721670825" watchObservedRunningTime="2026-05-25 11:17:19.438987358 +0000 UTC m=+7.435162173"
May 25 11:17:19.439500 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.439358    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-k7sb2"]
May 25 11:17:19.442595 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.442573    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:19.442718 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:19.442648    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:19.530217 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.530181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0bac8d4-5c32-4d69-9b35-2d0200701bea-kubelet-config\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:19.530365 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.530236    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:19.530365 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.530338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0bac8d4-5c32-4d69-9b35-2d0200701bea-dbus\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:19.565643 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.565453    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:19.565643 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:19.565603    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:19.631577 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.631538    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0bac8d4-5c32-4d69-9b35-2d0200701bea-kubelet-config\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:19.631739 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.631598    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:19.631739 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.631657    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0bac8d4-5c32-4d69-9b35-2d0200701bea-dbus\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:19.631935 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.631876    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0bac8d4-5c32-4d69-9b35-2d0200701bea-dbus\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:19.631989 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:19.631935    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0bac8d4-5c32-4d69-9b35-2d0200701bea-kubelet-config\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:19.632043 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:19.632034    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:19.632093 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:19.632088    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret podName:f0bac8d4-5c32-4d69-9b35-2d0200701bea nodeName:}" failed. No retries permitted until 2026-05-25 11:17:20.132070514 +0000 UTC m=+8.128245312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret") pod "global-pull-secret-syncer-k7sb2" (UID: "f0bac8d4-5c32-4d69-9b35-2d0200701bea") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:20.136163 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:20.136064    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:20.136344 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:20.136183    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:20.136344 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:20.136261    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret podName:f0bac8d4-5c32-4d69-9b35-2d0200701bea nodeName:}" failed. No retries permitted until 2026-05-25 11:17:21.136241383 +0000 UTC m=+9.132416190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret") pod "global-pull-secret-syncer-k7sb2" (UID: "f0bac8d4-5c32-4d69-9b35-2d0200701bea") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:20.565967 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:20.565935    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:20.566395 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:20.566087    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:21.146180 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:21.146144    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:21.146364 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:21.146281    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:21.146364 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:21.146339    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret podName:f0bac8d4-5c32-4d69-9b35-2d0200701bea nodeName:}" failed. No retries permitted until 2026-05-25 11:17:23.146320751 +0000 UTC m=+11.142495546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret") pod "global-pull-secret-syncer-k7sb2" (UID: "f0bac8d4-5c32-4d69-9b35-2d0200701bea") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:21.565324 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:21.565291    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:21.565488 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:21.565433    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:21.566623 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:21.566337    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:21.566623 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:21.566535    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:22.154340 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:22.154301    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:22.154528 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:22.154447    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:22.154598 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:22.154552    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs podName:c8d62b8b-fa29-494d-a37a-3436347370ed nodeName:}" failed. No retries permitted until 2026-05-25 11:17:30.154531127 +0000 UTC m=+18.150705931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs") pod "network-metrics-daemon-z2l9j" (UID: "c8d62b8b-fa29-494d-a37a-3436347370ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:22.254905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:22.254866    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:22.255073 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:22.255024    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:22.255073 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:22.255046    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:22.255073 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:22.255058    2569 projected.go:194] Error preparing data for projected volume kube-api-access-9nw7d for pod openshift-network-diagnostics/network-check-target-gzg8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:22.255188 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:22.255113    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d podName:607c3221-179d-44be-81d0-930ca62118ad nodeName:}" failed. No retries permitted until 2026-05-25 11:17:30.255095898 +0000 UTC m=+18.251270712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nw7d" (UniqueName: "kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d") pod "network-check-target-gzg8z" (UID: "607c3221-179d-44be-81d0-930ca62118ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:22.566385 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:22.566355    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:22.566564 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:22.566462    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:23.162288 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:23.162247    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:23.162743 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:23.162381    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:23.162743 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:23.162436    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret podName:f0bac8d4-5c32-4d69-9b35-2d0200701bea nodeName:}" failed. No retries permitted until 2026-05-25 11:17:27.162422534 +0000 UTC m=+15.158597338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret") pod "global-pull-secret-syncer-k7sb2" (UID: "f0bac8d4-5c32-4d69-9b35-2d0200701bea") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:23.565142 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:23.565106    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:23.565300 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:23.565106    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:23.565300 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:23.565239    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:23.565418 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:23.565371    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:24.565637 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:24.565601    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:24.566070 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:24.565735    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:25.565572 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:25.565538    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:25.565795 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:25.565626    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:25.565795 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:25.565758    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:25.566212 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:25.565906    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:26.565520 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:26.565490    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:26.565840 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:26.565608    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:27.193072 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:27.193040    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:27.193248 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:27.193156    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:27.193248 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:27.193211    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret podName:f0bac8d4-5c32-4d69-9b35-2d0200701bea nodeName:}" failed. No retries permitted until 2026-05-25 11:17:35.193198388 +0000 UTC m=+23.189373179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret") pod "global-pull-secret-syncer-k7sb2" (UID: "f0bac8d4-5c32-4d69-9b35-2d0200701bea") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:27.566064 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:27.566024    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:27.566064 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:27.566057    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:27.566448 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:27.566152    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:27.566448 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:27.566280    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:28.565982 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:28.565950    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:28.566154 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:28.566081    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:29.565945 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:29.565911    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:29.566099 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:29.566038    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:29.566152 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:29.566114    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:29.566233 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:29.566212    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:30.214869 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:30.214834    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:30.215037 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:30.214994    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:30.215080 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:30.215064    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs podName:c8d62b8b-fa29-494d-a37a-3436347370ed nodeName:}" failed. No retries permitted until 2026-05-25 11:17:46.215043589 +0000 UTC m=+34.211218392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs") pod "network-metrics-daemon-z2l9j" (UID: "c8d62b8b-fa29-494d-a37a-3436347370ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:30.315709 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:30.315669    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:30.315898 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:30.315856    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:30.315898 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:30.315876    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:30.315898 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:30.315887    2569 projected.go:194] Error preparing data for projected volume kube-api-access-9nw7d for pod openshift-network-diagnostics/network-check-target-gzg8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:30.316051 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:30.315938    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d podName:607c3221-179d-44be-81d0-930ca62118ad nodeName:}" failed. No retries permitted until 2026-05-25 11:17:46.315922811 +0000 UTC m=+34.312097607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nw7d" (UniqueName: "kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d") pod "network-check-target-gzg8z" (UID: "607c3221-179d-44be-81d0-930ca62118ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:30.566008 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:30.565972    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:30.566183 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:30.566119    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:31.566017 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:31.565983    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:31.566456 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:31.565990    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:31.566456 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:31.566090    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:31.566456 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:31.566170    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:32.566623 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:32.566588    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:32.567061 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:32.566677    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:33.565420 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.565090    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:33.565536 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.565156    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:33.565595 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:33.565523    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:33.565665 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:33.565648    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:33.743018 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.742982    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sg6vv" event={"ID":"9fa1e4cb-c587-401c-8dc7-a6bc27de70d3","Type":"ContainerStarted","Data":"3c440f7b8b7a0375be26603ddf2acfac5e325840a9e862200c55d9bbbbef9925"}
May 25 11:17:33.744342 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.744311    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jfr78" event={"ID":"295056bc-0e37-4116-bbd8-00f40557f5d5","Type":"ContainerStarted","Data":"a57f2f5ca0df1ba4e6ef05c3fe2c49180ab941fa8421d6b9265c195e74b452e7"}
May 25 11:17:33.745875 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.745853    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx" event={"ID":"fdba9b03-f691-4b5e-840c-2bfb07c72094","Type":"ContainerStarted","Data":"b719dcb7affa4665d9369f0518aba56d5a097f5863ba0b71fe5aa2016810f7cd"}
May 25 11:17:33.747260 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.747238    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hjknq" event={"ID":"b3126fb3-2cba-43c5-9244-4cc915eac03d","Type":"ContainerStarted","Data":"a7d76aa56dd051988d19fadc377a8eb6364d0a6fbce764f05f53700f958fb038"}
May 25 11:17:33.748728 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.748691    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vlsbt" event={"ID":"9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605","Type":"ContainerStarted","Data":"9289c1753612d140e5f08c047aa0c98ffa7894ee1ef9c2aa15cf1c85f4436dad"}
May 25 11:17:33.751124 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.751107    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:17:33.751480 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.751457    2569 generic.go:358] "Generic (PLEG): container finished" podID="f67dfd38-152e-40bf-beb0-430505aa58fe" containerID="e7ed1321ba973e39ebf37430ae0fd61fc5ce828f49ecd0f8d0801f7b20ef78c7" exitCode=1
May 25 11:17:33.751566 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.751479    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerStarted","Data":"f8a9aac133c3cc8f055b8749e7804780be47544d76b5193051244bfa99939af3"}
May 25 11:17:33.751566 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.751506    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerStarted","Data":"0bf93eadf4bd7a22806428a293c573b7d400718f3f2ef13bfd9cb36a12cc23c3"}
May 25 11:17:33.751566 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.751521    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerStarted","Data":"a5d31d093b47a6b854fce7c4c6389e771f5006dd1e843de77612915039fa1f52"}
May 25 11:17:33.751566 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.751535    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerDied","Data":"e7ed1321ba973e39ebf37430ae0fd61fc5ce828f49ecd0f8d0801f7b20ef78c7"}
May 25 11:17:33.751566 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.751553    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerStarted","Data":"20dc9f4919bef2860f84a5cf67fd22a78a8cadf1ee1fc7111e47b13c0aa6c355"}
May 25 11:17:33.752899 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.752881    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwq6c" event={"ID":"b32b422f-cd6e-4bcb-a52a-9c7b3e895676","Type":"ContainerStarted","Data":"ee2f60dbaba547f96d76be66e55f7cb56cc8a6023844137e3777600d6db22ddb"}
May 25 11:17:33.754274 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.754252    2569 generic.go:358] "Generic (PLEG): container finished" podID="9b5d0970-d81c-4ffb-9e1d-d2be86769fb9" containerID="83b0823175300a4d6b134d4ad91e6e76a73788313df16c07a2bd53faa404dfce" exitCode=0
May 25 11:17:33.754372 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.754286    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5mdm" event={"ID":"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9","Type":"ContainerDied","Data":"83b0823175300a4d6b134d4ad91e6e76a73788313df16c07a2bd53faa404dfce"}
May 25 11:17:33.757754 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.757710    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-sg6vv" podStartSLOduration=3.913228745 podStartE2EDuration="21.757700668s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:15.060438423 +0000 UTC m=+3.056613229" lastFinishedPulling="2026-05-25 11:17:32.904910347 +0000 UTC m=+20.901085152" observedRunningTime="2026-05-25 11:17:33.75758468 +0000 UTC m=+21.753759495" watchObservedRunningTime="2026-05-25 11:17:33.757700668 +0000 UTC m=+21.753875480"
May 25 11:17:33.779043 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.779003    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-jfr78" podStartSLOduration=3.8763656810000002 podStartE2EDuration="21.778989011s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:15.060194736 +0000 UTC m=+3.056369527" lastFinishedPulling="2026-05-25 11:17:32.962818041 +0000 UTC m=+20.958992857" observedRunningTime="2026-05-25 11:17:33.77887726 +0000 UTC m=+21.775052073" watchObservedRunningTime="2026-05-25 11:17:33.778989011 +0000 UTC m=+21.775163825"
May 25 11:17:33.795322 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.795280    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hjknq" podStartSLOduration=3.945971478 podStartE2EDuration="21.795266263s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:15.055308214 +0000 UTC m=+3.051483006" lastFinishedPulling="2026-05-25 11:17:32.904602982 +0000 UTC m=+20.900777791" observedRunningTime="2026-05-25 11:17:33.794998411 +0000 UTC m=+21.791173224" watchObservedRunningTime="2026-05-25 11:17:33.795266263 +0000 UTC m=+21.791441077"
May 25 11:17:33.809588 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.809553    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nwq6c" podStartSLOduration=12.073631684 podStartE2EDuration="21.809543368s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:15.068045551 +0000 UTC m=+3.064220341" lastFinishedPulling="2026-05-25 11:17:24.803957219 +0000 UTC m=+12.800132025" observedRunningTime="2026-05-25 11:17:33.809130051 +0000 UTC m=+21.805304865" watchObservedRunningTime="2026-05-25 11:17:33.809543368 +0000 UTC m=+21.805718180"
May 25 11:17:33.861858 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.861817    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vlsbt" podStartSLOduration=4.00783216 podStartE2EDuration="21.861804285s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:15.05371339 +0000 UTC m=+3.049888193" lastFinishedPulling="2026-05-25 11:17:32.907685523 +0000 UTC m=+20.903860318" observedRunningTime="2026-05-25 11:17:33.861375286 +0000 UTC m=+21.857550099" watchObservedRunningTime="2026-05-25 11:17:33.861804285 +0000 UTC m=+21.857979097"
May 25 11:17:33.991517 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:33.991494    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 11:17:34.539946 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:34.539832    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T11:17:33.991512428Z","UUID":"5b91f06b-20bc-4572-ae73-2384a17ccfe7","Handler":null,"Name":"","Endpoint":""}
May 25 11:17:34.541845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:34.541817    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 11:17:34.541845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:34.541845    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 11:17:34.566138 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:34.566109    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:34.566266 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:34.566238    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:34.757065 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:34.757032    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx" event={"ID":"fdba9b03-f691-4b5e-840c-2bfb07c72094","Type":"ContainerStarted","Data":"5e25ed5843a131a0c604b2a3b5d4a3bfd5c362f8a56a6504a3ce755c4442a4f9"}
May 25 11:17:34.758465 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:34.758431    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5tbtf" event={"ID":"28e6f486-3b90-4ab3-8088-d41e46546740","Type":"ContainerStarted","Data":"59a98bc4baf253a9081cff3ab7f6120c53a60899698077002ade61fae2cbdde2"}
May 25 11:17:34.760890 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:34.760876    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:17:34.761197 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:34.761174    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerStarted","Data":"f1b30dfc0e0572d57c9aad08c9967573d8c8ea93a2e0527da49f41eda35fd5dd"}
May 25 11:17:34.773566 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:34.773523    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5tbtf" podStartSLOduration=4.964438023 podStartE2EDuration="22.77350944s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:15.064949562 +0000 UTC m=+3.061124354" lastFinishedPulling="2026-05-25 11:17:32.874020965 +0000 UTC m=+20.870195771" observedRunningTime="2026-05-25 11:17:34.773400417 +0000 UTC m=+22.769575230" watchObservedRunningTime="2026-05-25 11:17:34.77350944 +0000 UTC m=+22.769684254"
May 25 11:17:35.259892 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:35.259637    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:35.260042 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:35.259812    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:35.260100 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:35.260043    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret podName:f0bac8d4-5c32-4d69-9b35-2d0200701bea nodeName:}" failed. No retries permitted until 2026-05-25 11:17:51.260019178 +0000 UTC m=+39.256193983 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret") pod "global-pull-secret-syncer-k7sb2" (UID: "f0bac8d4-5c32-4d69-9b35-2d0200701bea") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:35.565399 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:35.565341    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:35.565570 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:35.565351    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:35.565570 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:35.565444    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:35.565570 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:35.565550    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:35.764682 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:35.764646    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx" event={"ID":"fdba9b03-f691-4b5e-840c-2bfb07c72094","Type":"ContainerStarted","Data":"cfb9e70e22be57e1dec1233aa7863d0b262f0a701f3181e68afa15dcdab2c19e"}
May 25 11:17:35.785023 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:35.784969    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dhnhx" podStartSLOduration=3.808100336 podStartE2EDuration="23.784955669s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:15.057966898 +0000 UTC m=+3.054141694" lastFinishedPulling="2026-05-25 11:17:35.034822221 +0000 UTC m=+23.030997027" observedRunningTime="2026-05-25 11:17:35.784815502 +0000 UTC m=+23.780990315" watchObservedRunningTime="2026-05-25 11:17:35.784955669 +0000 UTC m=+23.781130519"
May 25 11:17:35.814084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:35.814062    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:35.819425 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:35.819366    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:36.368401 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:36.368367    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:36.368984 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:36.368960    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hjknq"
May 25 11:17:36.568627 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:36.568593    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:36.568825 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:36.568690    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:36.769227 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:36.769197    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:17:36.769713 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:36.769553    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerStarted","Data":"c9f55aab916f6b6e3b290aea463414b3541eb73b0f3fd8bfca9bbf378ddc23a6"}
May 25 11:17:37.565398 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:37.565366    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:37.565558 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:37.565382    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:37.565558 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:37.565503    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:37.565657 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:37.565586    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:38.566218 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:38.566047    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:38.566798 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:38.566306    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:38.774434 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:38.774396    2569 generic.go:358] "Generic (PLEG): container finished" podID="9b5d0970-d81c-4ffb-9e1d-d2be86769fb9" containerID="f780210b78d159e07eb76cfd471a4b7230e77b8b6c3f15233a7eadb0978c9125" exitCode=0
May 25 11:17:38.774579 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:38.774478    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5mdm" event={"ID":"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9","Type":"ContainerDied","Data":"f780210b78d159e07eb76cfd471a4b7230e77b8b6c3f15233a7eadb0978c9125"}
May 25 11:17:38.777546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:38.777529    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:17:38.777875 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:38.777850    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerStarted","Data":"cd5b24dfa7617f950e0f1660a1bec8813eb52db607549327bcb6114ed92a6338"}
May 25 11:17:38.778201 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:38.778186    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:38.778270 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:38.778211    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:38.778333 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:38.778318    2569 scope.go:117] "RemoveContainer" containerID="e7ed1321ba973e39ebf37430ae0fd61fc5ce828f49ecd0f8d0801f7b20ef78c7"
May 25 11:17:38.793407 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:38.793387    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:39.565344 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:39.565318    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:39.565521 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:39.565318    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:39.565521 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:39.565432    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:39.565612 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:39.565520    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:39.782873 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:39.782844    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:17:39.783279 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:39.783152    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" event={"ID":"f67dfd38-152e-40bf-beb0-430505aa58fe","Type":"ContainerStarted","Data":"6bce2832618a6e76cef06bfc503881a21c90d1cd48f7cd5a9a6b9e267e0a3151"}
May 25 11:17:39.783396 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:39.783378    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:39.798001 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:39.797973    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:17:39.820200 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:39.820116    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj" podStartSLOduration=9.953919634 podStartE2EDuration="27.820103586s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:15.061172406 +0000 UTC m=+3.057347197" lastFinishedPulling="2026-05-25 11:17:32.927356353 +0000 UTC m=+20.923531149" observedRunningTime="2026-05-25 11:17:39.818198834 +0000 UTC m=+27.814373669" watchObservedRunningTime="2026-05-25 11:17:39.820103586 +0000 UTC m=+27.816278399"
May 25 11:17:40.079935 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:40.079911    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z2l9j"]
May 25 11:17:40.080047 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:40.080007    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:40.080102 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:40.080086    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:40.083151 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:40.083129    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k7sb2"]
May 25 11:17:40.083262 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:40.083223    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:40.083380 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:40.083327    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:40.083807 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:40.083762    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gzg8z"]
May 25 11:17:40.083885 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:40.083875    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:40.083956 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:40.083940    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:40.786915 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:40.786874    2569 generic.go:358] "Generic (PLEG): container finished" podID="9b5d0970-d81c-4ffb-9e1d-d2be86769fb9" containerID="c83a5fa3b0d9a3c88b3bf59e689f51bfd80bdef2a08c5cfadf7a3fd0a9db9ee2" exitCode=0
May 25 11:17:40.787299 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:40.786941    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5mdm" event={"ID":"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9","Type":"ContainerDied","Data":"c83a5fa3b0d9a3c88b3bf59e689f51bfd80bdef2a08c5cfadf7a3fd0a9db9ee2"}
May 25 11:17:41.565204 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:41.565175    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:41.565369 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:41.565175    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:41.565369 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:41.565267    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:41.565369 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:41.565175    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:41.565369 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:41.565340    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:41.565533 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:41.565411    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:42.793615 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:42.793588    2569 generic.go:358] "Generic (PLEG): container finished" podID="9b5d0970-d81c-4ffb-9e1d-d2be86769fb9" containerID="2974cae20f6b4d92c214e4de37c226f8db5d5a5d6429725824c866be3a6d45dd" exitCode=0
May 25 11:17:42.794275 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:42.793651    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5mdm" event={"ID":"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9","Type":"ContainerDied","Data":"2974cae20f6b4d92c214e4de37c226f8db5d5a5d6429725824c866be3a6d45dd"}
May 25 11:17:43.565114 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:43.565070    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:43.565304 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:43.565181    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:43.565304 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:43.565078    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:43.565304 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:43.565070    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:43.565450 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:43.565322    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:43.565450 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:43.565411    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:45.565749 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:45.565716    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:45.566233 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:45.565716    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:45.566233 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:45.565716    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:45.566233 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:45.565898    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7sb2" podUID="f0bac8d4-5c32-4d69-9b35-2d0200701bea"
May 25 11:17:45.566233 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:45.565961    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2l9j" podUID="c8d62b8b-fa29-494d-a37a-3436347370ed"
May 25 11:17:45.566233 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:45.566043    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzg8z" podUID="607c3221-179d-44be-81d0-930ca62118ad"
May 25 11:17:46.249575 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.249543    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:46.249793 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:46.249692    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:46.249793 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:46.249753    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs podName:c8d62b8b-fa29-494d-a37a-3436347370ed nodeName:}" failed. No retries permitted until 2026-05-25 11:18:18.249736619 +0000 UTC m=+66.245911410 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs") pod "network-metrics-daemon-z2l9j" (UID: "c8d62b8b-fa29-494d-a37a-3436347370ed") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:46.350624 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.350586    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:46.350817 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:46.350736    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:46.350817 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:46.350756    2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:46.350817 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:46.350780    2569 projected.go:194] Error preparing data for projected volume kube-api-access-9nw7d for pod openshift-network-diagnostics/network-check-target-gzg8z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:46.350981 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:46.350833    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d podName:607c3221-179d-44be-81d0-930ca62118ad nodeName:}" failed. No retries permitted until 2026-05-25 11:18:18.350819612 +0000 UTC m=+66.346994402 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nw7d" (UniqueName: "kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d") pod "network-check-target-gzg8z" (UID: "607c3221-179d-44be-81d0-930ca62118ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:46.820204 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.820127    2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-158.ec2.internal" event="NodeReady"
May 25 11:17:46.820693 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.820269    2569 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 11:17:46.867086 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.867054    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d59fc58bc-55k7g"]
May 25 11:17:46.897711 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.897671    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gktpf"]
May 25 11:17:46.897886 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.897744    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:46.900384 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.900362    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 11:17:46.900607 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.900592    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 11:17:46.900689 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.900628    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gzhs6\""
May 25 11:17:46.900746 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.900628    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 11:17:46.906666 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.906646    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 11:17:46.911451 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.911433    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vlfq6"]
May 25 11:17:46.911613 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.911596    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gktpf"
May 25 11:17:46.914176 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.914158    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlbp4\""
May 25 11:17:46.914277 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.914154    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 11:17:46.914339 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.914272    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 11:17:46.929356 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.929342    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d59fc58bc-55k7g"]
May 25 11:17:46.929451 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.929361    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlfq6"]
May 25 11:17:46.929451 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.929372    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gktpf"]
May 25 11:17:46.929532 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.929459    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:46.931877 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.931860    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 11:17:46.931994 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.931975    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kttpg\""
May 25 11:17:46.932107 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.931862    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 11:17:46.932107 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:46.931912    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 11:17:47.055133 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055100    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-config-volume\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.055133 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-tmp-dir\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.055393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055188    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-image-registry-private-configuration\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.055393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055221    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:47.055393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055246    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-installation-pull-secrets\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.055393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055282    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.055393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055297    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86547d63-5a65-4222-b6d0-0407b98b9b34-ca-trust-extracted\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.055393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.055393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-bound-sa-token\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.055668 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-certificates\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.055668 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055536    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-trusted-ca\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.055668 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsm77\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-kube-api-access-wsm77\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.055668 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhprn\" (UniqueName: \"kubernetes.io/projected/f2bbe64f-7412-4907-bd9f-658a5034f66c-kube-api-access-dhprn\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:47.055668 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.055618    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9gdg\" (UniqueName: \"kubernetes.io/projected/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-kube-api-access-p9gdg\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.156541 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156451    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-image-registry-private-configuration\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.156541 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156501    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:47.156541 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156532    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-installation-pull-secrets\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156595    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156621    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86547d63-5a65-4222-b6d0-0407b98b9b34-ca-trust-extracted\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.156627    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156649    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156672    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-bound-sa-token\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.156726    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert podName:f2bbe64f-7412-4907-bd9f-658a5034f66c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:47.656705458 +0000 UTC m=+35.652880268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert") pod "ingress-canary-vlfq6" (UID: "f2bbe64f-7412-4907-bd9f-658a5034f66c") : secret "canary-serving-cert" not found
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.156742    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.156760    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59fc58bc-55k7g: secret "image-registry-tls" not found
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156792    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-certificates\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.156823    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls podName:86547d63-5a65-4222-b6d0-0407b98b9b34 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:47.656808167 +0000 UTC m=+35.652982974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls") pod "image-registry-6d59fc58bc-55k7g" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34") : secret "image-registry-tls" not found
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156846    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-trusted-ca\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156875    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wsm77\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-kube-api-access-wsm77\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156898    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dhprn\" (UniqueName: \"kubernetes.io/projected/f2bbe64f-7412-4907-bd9f-658a5034f66c-kube-api-access-dhprn\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:47.156920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156923    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p9gdg\" (UniqueName: \"kubernetes.io/projected/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-kube-api-access-p9gdg\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.157516 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156947    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-config-volume\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.157516 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.156968    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-tmp-dir\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.157516 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.157055    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86547d63-5a65-4222-b6d0-0407b98b9b34-ca-trust-extracted\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.157516 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.156849    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:47.157516 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.157135    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls podName:ebf5f84b-f809-49d2-8c90-bdedd5b862e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:47.6571208 +0000 UTC m=+35.653295607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls") pod "dns-default-gktpf" (UID: "ebf5f84b-f809-49d2-8c90-bdedd5b862e8") : secret "dns-default-metrics-tls" not found
May 25 11:17:47.157678 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.157636    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-certificates\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.157678 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.157647    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-config-volume\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.157875 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.157860    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-tmp-dir\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.157921 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.157894    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-trusted-ca\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.161079 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.161055    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-installation-pull-secrets\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.161223 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.161115    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-image-registry-private-configuration\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.175905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.175728    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsm77\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-kube-api-access-wsm77\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.176024 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.175734    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhprn\" (UniqueName: \"kubernetes.io/projected/f2bbe64f-7412-4907-bd9f-658a5034f66c-kube-api-access-dhprn\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:47.177948 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.177927    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-bound-sa-token\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.178067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.178052    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9gdg\" (UniqueName: \"kubernetes.io/projected/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-kube-api-access-p9gdg\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.566133 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.566101    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:47.566338 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.566154    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:17:47.566431 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.566333    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:17:47.569113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.569095    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 11:17:47.570188 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.570167    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:17:47.570301 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.570187    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf8xr\""
May 25 11:17:47.570301 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.570268    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:17:47.570415 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.570320    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x64h9\""
May 25 11:17:47.570466 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.570438    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:17:47.662282 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.662248    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:47.662524 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.662337    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:47.662524 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:47.662374    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:47.662630 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.662527    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:47.662630 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.662596    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls podName:ebf5f84b-f809-49d2-8c90-bdedd5b862e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.66256886 +0000 UTC m=+36.658743663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls") pod "dns-default-gktpf" (UID: "ebf5f84b-f809-49d2-8c90-bdedd5b862e8") : secret "dns-default-metrics-tls" not found
May 25 11:17:47.664797 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.662898    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:47.664797 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.662919    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59fc58bc-55k7g: secret "image-registry-tls" not found
May 25 11:17:47.664797 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.662985    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls podName:86547d63-5a65-4222-b6d0-0407b98b9b34 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.662967247 +0000 UTC m=+36.659142044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls") pod "image-registry-6d59fc58bc-55k7g" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34") : secret "image-registry-tls" not found
May 25 11:17:47.664797 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.663062    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:47.664797 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:47.663104    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert podName:f2bbe64f-7412-4907-bd9f-658a5034f66c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.663092163 +0000 UTC m=+36.659266969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert") pod "ingress-canary-vlfq6" (UID: "f2bbe64f-7412-4907-bd9f-658a5034f66c") : secret "canary-serving-cert" not found
May 25 11:17:48.670830 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:48.670746    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:48.670830 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:48.670826    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:48.671238 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:48.670848    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:48.671238 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:48.670882    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:48.671238 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:48.670949    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert podName:f2bbe64f-7412-4907-bd9f-658a5034f66c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:50.670932598 +0000 UTC m=+38.667107389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert") pod "ingress-canary-vlfq6" (UID: "f2bbe64f-7412-4907-bd9f-658a5034f66c") : secret "canary-serving-cert" not found
May 25 11:17:48.671238 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:48.670954    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:48.671238 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:48.670958    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:48.671238 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:48.671010    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls podName:ebf5f84b-f809-49d2-8c90-bdedd5b862e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:50.670998454 +0000 UTC m=+38.667173244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls") pod "dns-default-gktpf" (UID: "ebf5f84b-f809-49d2-8c90-bdedd5b862e8") : secret "dns-default-metrics-tls" not found
May 25 11:17:48.671238 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:48.671015    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59fc58bc-55k7g: secret "image-registry-tls" not found
May 25 11:17:48.671238 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:48.671060    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls podName:86547d63-5a65-4222-b6d0-0407b98b9b34 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:50.671047522 +0000 UTC m=+38.667222319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls") pod "image-registry-6d59fc58bc-55k7g" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34") : secret "image-registry-tls" not found
May 25 11:17:48.807527 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:48.807497    2569 generic.go:358] "Generic (PLEG): container finished" podID="9b5d0970-d81c-4ffb-9e1d-d2be86769fb9" containerID="33cb5d9f1303fade1b2c95ca7517a6e791fedbf481e4e3b05b365b92bda81fc2" exitCode=0
May 25 11:17:48.807665 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:48.807557    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5mdm" event={"ID":"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9","Type":"ContainerDied","Data":"33cb5d9f1303fade1b2c95ca7517a6e791fedbf481e4e3b05b365b92bda81fc2"}
May 25 11:17:49.811467 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:49.811434    2569 generic.go:358] "Generic (PLEG): container finished" podID="9b5d0970-d81c-4ffb-9e1d-d2be86769fb9" containerID="be86e4cffbdb3005d669cf0fa7c26f587cf8e2846f6da9b1c31403a3525c61b9" exitCode=0
May 25 11:17:49.811813 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:49.811500    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5mdm" event={"ID":"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9","Type":"ContainerDied","Data":"be86e4cffbdb3005d669cf0fa7c26f587cf8e2846f6da9b1c31403a3525c61b9"}
May 25 11:17:50.685384 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:50.685338    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:50.685384 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:50.685404    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:50.685603 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:50.685432    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:50.685603 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:50.685490    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:50.685603 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:50.685526    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:50.685603 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:50.685553    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert podName:f2bbe64f-7412-4907-bd9f-658a5034f66c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:54.68553895 +0000 UTC m=+42.681713741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert") pod "ingress-canary-vlfq6" (UID: "f2bbe64f-7412-4907-bd9f-658a5034f66c") : secret "canary-serving-cert" not found
May 25 11:17:50.685603 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:50.685557    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:50.685603 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:50.685572    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59fc58bc-55k7g: secret "image-registry-tls" not found
May 25 11:17:50.685603 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:50.685574    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls podName:ebf5f84b-f809-49d2-8c90-bdedd5b862e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:54.685559265 +0000 UTC m=+42.681734057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls") pod "dns-default-gktpf" (UID: "ebf5f84b-f809-49d2-8c90-bdedd5b862e8") : secret "dns-default-metrics-tls" not found
May 25 11:17:50.685603 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:50.685606    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls podName:86547d63-5a65-4222-b6d0-0407b98b9b34 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:54.685596708 +0000 UTC m=+42.681771512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls") pod "image-registry-6d59fc58bc-55k7g" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34") : secret "image-registry-tls" not found
May 25 11:17:50.816111 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:50.816074    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5mdm" event={"ID":"9b5d0970-d81c-4ffb-9e1d-d2be86769fb9","Type":"ContainerStarted","Data":"08287428d87085076b969a3241ebf468950bce9d56d63fcabcd6c2ff94003083"}
May 25 11:17:50.857172 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:50.857130    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b5mdm" podStartSLOduration=5.612407999 podStartE2EDuration="38.857116991s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:15.062311076 +0000 UTC m=+3.058485866" lastFinishedPulling="2026-05-25 11:17:48.307020067 +0000 UTC m=+36.303194858" observedRunningTime="2026-05-25 11:17:50.854887106 +0000 UTC m=+38.851061919" watchObservedRunningTime="2026-05-25 11:17:50.857116991 +0000 UTC m=+38.853291803"
May 25 11:17:51.289893 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:51.289854    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:51.293075 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:51.293050    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bac8d4-5c32-4d69-9b35-2d0200701bea-original-pull-secret\") pod \"global-pull-secret-syncer-k7sb2\" (UID: \"f0bac8d4-5c32-4d69-9b35-2d0200701bea\") " pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:51.490199 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:51.490170    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7sb2"
May 25 11:17:51.621719 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:51.621582    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k7sb2"]
May 25 11:17:51.624358 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:51.624332    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bac8d4_5c32_4d69_9b35_2d0200701bea.slice/crio-a4e00ffcbb8b81715542e36bae6b604651fef2fafc0a8b7e62ee7fde735858cc WatchSource:0}: Error finding container a4e00ffcbb8b81715542e36bae6b604651fef2fafc0a8b7e62ee7fde735858cc: Status 404 returned error can't find the container with id a4e00ffcbb8b81715542e36bae6b604651fef2fafc0a8b7e62ee7fde735858cc
May 25 11:17:51.818647 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:51.818568    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k7sb2" event={"ID":"f0bac8d4-5c32-4d69-9b35-2d0200701bea","Type":"ContainerStarted","Data":"a4e00ffcbb8b81715542e36bae6b604651fef2fafc0a8b7e62ee7fde735858cc"}
May 25 11:17:52.334664 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.334608    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs"]
May 25 11:17:52.338090 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.338066    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs"
May 25 11:17:52.340545 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.340521    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 25 11:17:52.341671 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.341650    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 25 11:17:52.341763 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.341694    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hhkcw\""
May 25 11:17:52.355268 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.355242    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs"]
May 25 11:17:52.401016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.400990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b66p\" (UniqueName: \"kubernetes.io/projected/2f1e16c7-8531-460c-b2d9-013449b660fa-kube-api-access-6b66p\") pod \"migrator-b985f5d89-jj8fs\" (UID: \"2f1e16c7-8531-460c-b2d9-013449b660fa\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs"
May 25 11:17:52.501544 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.501504    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6b66p\" (UniqueName: \"kubernetes.io/projected/2f1e16c7-8531-460c-b2d9-013449b660fa-kube-api-access-6b66p\") pod \"migrator-b985f5d89-jj8fs\" (UID: \"2f1e16c7-8531-460c-b2d9-013449b660fa\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs"
May 25 11:17:52.509579 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.509554    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b66p\" (UniqueName: \"kubernetes.io/projected/2f1e16c7-8531-460c-b2d9-013449b660fa-kube-api-access-6b66p\") pod \"migrator-b985f5d89-jj8fs\" (UID: \"2f1e16c7-8531-460c-b2d9-013449b660fa\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs"
May 25 11:17:52.649230 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.649138    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs"
May 25 11:17:52.792763 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.792725    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs"]
May 25 11:17:52.796541 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:17:52.796511    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1e16c7_8531_460c_b2d9_013449b660fa.slice/crio-fe681cf43901d49f7d686645ffb164300d8066eef5e8852ff102152d15ac87ec WatchSource:0}: Error finding container fe681cf43901d49f7d686645ffb164300d8066eef5e8852ff102152d15ac87ec: Status 404 returned error can't find the container with id fe681cf43901d49f7d686645ffb164300d8066eef5e8852ff102152d15ac87ec
May 25 11:17:52.821383 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:52.821358    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs" event={"ID":"2f1e16c7-8531-460c-b2d9-013449b660fa","Type":"ContainerStarted","Data":"fe681cf43901d49f7d686645ffb164300d8066eef5e8852ff102152d15ac87ec"}
May 25 11:17:53.930521 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:53.930498    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sg6vv_9fa1e4cb-c587-401c-8dc7-a6bc27de70d3/dns-node-resolver/0.log"
May 25 11:17:54.720369 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:54.720330    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:17:54.720572 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:54.720405    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:17:54.720572 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:54.720430    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:17:54.720572 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:54.720451    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:54.720572 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:54.720528    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert podName:f2bbe64f-7412-4907-bd9f-658a5034f66c nodeName:}" failed. No retries permitted until 2026-05-25 11:18:02.720506664 +0000 UTC m=+50.716681468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert") pod "ingress-canary-vlfq6" (UID: "f2bbe64f-7412-4907-bd9f-658a5034f66c") : secret "canary-serving-cert" not found
May 25 11:17:54.720572 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:54.720561    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:54.720818 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:54.720583    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59fc58bc-55k7g: secret "image-registry-tls" not found
May 25 11:17:54.720818 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:54.720632    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls podName:86547d63-5a65-4222-b6d0-0407b98b9b34 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:02.720616857 +0000 UTC m=+50.716791655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls") pod "image-registry-6d59fc58bc-55k7g" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34") : secret "image-registry-tls" not found
May 25 11:17:54.720818 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:54.720565    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:54.720818 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:17:54.720699    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls podName:ebf5f84b-f809-49d2-8c90-bdedd5b862e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:02.720682763 +0000 UTC m=+50.716857570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls") pod "dns-default-gktpf" (UID: "ebf5f84b-f809-49d2-8c90-bdedd5b862e8") : secret "dns-default-metrics-tls" not found
May 25 11:17:54.826900 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:54.826866    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs" event={"ID":"2f1e16c7-8531-460c-b2d9-013449b660fa","Type":"ContainerStarted","Data":"88d8b66866235a9d3a784c244add141324d5440b16456f71bd7ea87b2b1ab386"}
May 25 11:17:54.921685 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:54.921656    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwq6c_b32b422f-cd6e-4bcb-a52a-9c7b3e895676/node-ca/0.log"
May 25 11:17:55.829869 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:55.829829    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k7sb2" event={"ID":"f0bac8d4-5c32-4d69-9b35-2d0200701bea","Type":"ContainerStarted","Data":"24e2a66fea3a5e3072389099437642e7fc67d18ae86c7dff3bb95953b0cacbd9"}
May 25 11:17:55.831292 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:55.831268    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs" event={"ID":"2f1e16c7-8531-460c-b2d9-013449b660fa","Type":"ContainerStarted","Data":"9e09cdbddb7674f04a662c05d9989a6c91dee4032b39a4a19caabb2f0b8fb112"}
May 25 11:17:55.847866 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:55.847802    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k7sb2" podStartSLOduration=33.228126617 podStartE2EDuration="36.847790025s" podCreationTimestamp="2026-05-25 11:17:19 +0000 UTC" firstStartedPulling="2026-05-25 11:17:51.626021308 +0000 UTC m=+39.622196098" lastFinishedPulling="2026-05-25 11:17:55.245684712 +0000 UTC m=+43.241859506" observedRunningTime="2026-05-25 11:17:55.847433736 +0000 UTC m=+43.843608551" watchObservedRunningTime="2026-05-25 11:17:55.847790025 +0000 UTC m=+43.843964832"
May 25 11:17:55.868019 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:17:55.867981    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-jj8fs" podStartSLOduration=2.840203743 podStartE2EDuration="3.867970563s" podCreationTimestamp="2026-05-25 11:17:52 +0000 UTC" firstStartedPulling="2026-05-25 11:17:52.798963387 +0000 UTC m=+40.795138178" lastFinishedPulling="2026-05-25 11:17:53.826730204 +0000 UTC m=+41.822904998" observedRunningTime="2026-05-25 11:17:55.86708416 +0000 UTC m=+43.863258993" watchObservedRunningTime="2026-05-25 11:17:55.867970563 +0000 UTC m=+43.864145375"
May 25 11:18:02.781264 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:02.781223    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:18:02.781264 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:02.781267    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:18:02.781696 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:02.781382    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:02.781696 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:02.781386    2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:02.781696 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:02.781408    2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59fc58bc-55k7g: secret "image-registry-tls" not found
May 25 11:18:02.781696 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:02.781426    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:18:02.781696 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:02.781442    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls podName:ebf5f84b-f809-49d2-8c90-bdedd5b862e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:18.781423525 +0000 UTC m=+66.777598332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls") pod "dns-default-gktpf" (UID: "ebf5f84b-f809-49d2-8c90-bdedd5b862e8") : secret "dns-default-metrics-tls" not found
May 25 11:18:02.781696 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:02.781463    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls podName:86547d63-5a65-4222-b6d0-0407b98b9b34 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:18.781453717 +0000 UTC m=+66.777628510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls") pod "image-registry-6d59fc58bc-55k7g" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34") : secret "image-registry-tls" not found
May 25 11:18:02.781696 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:02.781496    2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:02.781696 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:02.781540    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert podName:f2bbe64f-7412-4907-bd9f-658a5034f66c nodeName:}" failed. No retries permitted until 2026-05-25 11:18:18.781527716 +0000 UTC m=+66.777702525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert") pod "ingress-canary-vlfq6" (UID: "f2bbe64f-7412-4907-bd9f-658a5034f66c") : secret "canary-serving-cert" not found
May 25 11:18:11.801619 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:11.801584    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbtgj"
May 25 11:18:18.287384 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.287345    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:18:18.289826 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.289807    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:18:18.299890 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.299862    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d62b8b-fa29-494d-a37a-3436347370ed-metrics-certs\") pod \"network-metrics-daemon-z2l9j\" (UID: \"c8d62b8b-fa29-494d-a37a-3436347370ed\") " pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:18:18.388151 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.388123    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:18:18.402598 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.402577    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:18:18.411840 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.411820    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:18:18.415417 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.415398    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"]
May 25 11:18:18.418063 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.418047    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.421112 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.421088    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nw7d\" (UniqueName: \"kubernetes.io/projected/607c3221-179d-44be-81d0-930ca62118ad-kube-api-access-9nw7d\") pod \"network-check-target-gzg8z\" (UID: \"607c3221-179d-44be-81d0-930ca62118ad\") " pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:18:18.422032 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.422015    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 25 11:18:18.424727 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.424701    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 25 11:18:18.424933 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.424920    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 25 11:18:18.425345 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.425330    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 25 11:18:18.425528 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.425512    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 25 11:18:18.425616 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.425529    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 25 11:18:18.425616 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.425516    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 25 11:18:18.470573 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.470551    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"]
May 25 11:18:18.488510 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.488483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/91347ebc-aca7-4076-a5d2-487b1f433bb1-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.488599 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.488516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7q6l\" (UniqueName: \"kubernetes.io/projected/91347ebc-aca7-4076-a5d2-487b1f433bb1-kube-api-access-x7q6l\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.488599 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.488587    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf8xr\""
May 25 11:18:18.488686 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.488624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-ca\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.488686 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.488645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-hub\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.488686 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.488665    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.488686 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.488682    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.494877 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.494866    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:18:18.521861 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.521838    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x64h9\""
May 25 11:18:18.528092 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.528071    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2l9j"
May 25 11:18:18.591422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.590729    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/91347ebc-aca7-4076-a5d2-487b1f433bb1-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.591422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.590802    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x7q6l\" (UniqueName: \"kubernetes.io/projected/91347ebc-aca7-4076-a5d2-487b1f433bb1-kube-api-access-x7q6l\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.591422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.590918    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-ca\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.591422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.590945    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-hub\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.591422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.590975    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.591422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.591004    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.592344 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.592316    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/91347ebc-aca7-4076-a5d2-487b1f433bb1-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.594244 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.594198    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.594451 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.594426    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-hub\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.595314 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.595288    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.595601 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.595569    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/91347ebc-aca7-4076-a5d2-487b1f433bb1-ca\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.607483 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.607375    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d59fc58bc-55k7g"]
May 25 11:18:18.607638 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:18.607609    2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g" podUID="86547d63-5a65-4222-b6d0-0407b98b9b34"
May 25 11:18:18.609810 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.609790    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7q6l\" (UniqueName: \"kubernetes.io/projected/91347ebc-aca7-4076-a5d2-487b1f433bb1-kube-api-access-x7q6l\") pod \"cluster-proxy-proxy-agent-54647fd587-wwkzp\" (UID: \"91347ebc-aca7-4076-a5d2-487b1f433bb1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.669372 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.669336    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7qqxf"]
May 25 11:18:18.671878 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.671862    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.681021 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.681002    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gzg8z"]
May 25 11:18:18.682355 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:18:18.682333    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607c3221_179d_44be_81d0_930ca62118ad.slice/crio-1be759f3fcfcdfd9a3c2ded9755ed1f4b5026b7d43bb6f0ec98a826883e338e0 WatchSource:0}: Error finding container 1be759f3fcfcdfd9a3c2ded9755ed1f4b5026b7d43bb6f0ec98a826883e338e0: Status 404 returned error can't find the container with id 1be759f3fcfcdfd9a3c2ded9755ed1f4b5026b7d43bb6f0ec98a826883e338e0
May 25 11:18:18.693885 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.693865    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 11:18:18.721922 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.721902    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 11:18:18.722866 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.722845    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qqxf"]
May 25 11:18:18.725232 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.725216    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 11:18:18.730669 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.730653    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 11:18:18.731052 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.731033    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xd7cl\""
May 25 11:18:18.739555 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.739540    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"
May 25 11:18:18.777213 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.777190    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z2l9j"]
May 25 11:18:18.779101 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:18:18.779074    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d62b8b_fa29_494d_a37a_3436347370ed.slice/crio-550db79abf17e60633208fc4b43987290dc6c73cafdf7a55449756690a366e73 WatchSource:0}: Error finding container 550db79abf17e60633208fc4b43987290dc6c73cafdf7a55449756690a366e73: Status 404 returned error can't find the container with id 550db79abf17e60633208fc4b43987290dc6c73cafdf7a55449756690a366e73
May 25 11:18:18.793113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.792378    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-data-volume\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.793113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.792433    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4vd\" (UniqueName: \"kubernetes.io/projected/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-kube-api-access-8k4vd\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.793113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.792474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-crio-socket\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.793113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.792514    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:18:18.793113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.792560    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:18:18.793113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.792587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.793113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.792617    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:18:18.793113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.792665    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.796627 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.796600    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf5f84b-f809-49d2-8c90-bdedd5b862e8-metrics-tls\") pod \"dns-default-gktpf\" (UID: \"ebf5f84b-f809-49d2-8c90-bdedd5b862e8\") " pod="openshift-dns/dns-default-gktpf"
May 25 11:18:18.797382 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.797363    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2bbe64f-7412-4907-bd9f-658a5034f66c-cert\") pod \"ingress-canary-vlfq6\" (UID: \"f2bbe64f-7412-4907-bd9f-658a5034f66c\") " pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:18:18.798195 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.798176    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"image-registry-6d59fc58bc-55k7g\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") " pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:18:18.873162 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.873123    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2l9j" event={"ID":"c8d62b8b-fa29-494d-a37a-3436347370ed","Type":"ContainerStarted","Data":"550db79abf17e60633208fc4b43987290dc6c73cafdf7a55449756690a366e73"}
May 25 11:18:18.874142 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.874124    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:18:18.874142 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.874132    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gzg8z" event={"ID":"607c3221-179d-44be-81d0-930ca62118ad","Type":"ContainerStarted","Data":"1be759f3fcfcdfd9a3c2ded9755ed1f4b5026b7d43bb6f0ec98a826883e338e0"}
May 25 11:18:18.877865 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.877852    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:18:18.893609 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.893590    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.893670 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.893628    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-data-volume\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.893670 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.893649    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4vd\" (UniqueName: \"kubernetes.io/projected/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-kube-api-access-8k4vd\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.893748 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.893673    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-crio-socket\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.893748 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.893719    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.893862 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.893847    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-crio-socket\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.893947 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.893931    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-data-volume\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.894222 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.894205    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.895812 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.895797    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.928682 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.928663    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4vd\" (UniqueName: \"kubernetes.io/projected/b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc-kube-api-access-8k4vd\") pod \"insights-runtime-extractor-7qqxf\" (UID: \"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc\") " pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.930919 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:18:18.930891    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91347ebc_aca7_4076_a5d2_487b1f433bb1.slice/crio-bd2256809437dfd178508d103cfb508b4454085e9f70a9c9bc6a05216d3bc9b3 WatchSource:0}: Error finding container bd2256809437dfd178508d103cfb508b4454085e9f70a9c9bc6a05216d3bc9b3: Status 404 returned error can't find the container with id bd2256809437dfd178508d103cfb508b4454085e9f70a9c9bc6a05216d3bc9b3
May 25 11:18:18.931398 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.931376    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp"]
May 25 11:18:18.980295 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.980276    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qqxf"
May 25 11:18:18.994427 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994406    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86547d63-5a65-4222-b6d0-0407b98b9b34-ca-trust-extracted\") pod \"86547d63-5a65-4222-b6d0-0407b98b9b34\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") "
May 25 11:18:18.994522 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994445    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-trusted-ca\") pod \"86547d63-5a65-4222-b6d0-0407b98b9b34\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") "
May 25 11:18:18.994522 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994493    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-image-registry-private-configuration\") pod \"86547d63-5a65-4222-b6d0-0407b98b9b34\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") "
May 25 11:18:18.994680 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994519    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-installation-pull-secrets\") pod \"86547d63-5a65-4222-b6d0-0407b98b9b34\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") "
May 25 11:18:18.994680 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994546    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") pod \"86547d63-5a65-4222-b6d0-0407b98b9b34\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") "
May 25 11:18:18.994680 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994580    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-certificates\") pod \"86547d63-5a65-4222-b6d0-0407b98b9b34\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") "
May 25 11:18:18.994680 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994622    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-bound-sa-token\") pod \"86547d63-5a65-4222-b6d0-0407b98b9b34\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") "
May 25 11:18:18.994680 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994639    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86547d63-5a65-4222-b6d0-0407b98b9b34-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "86547d63-5a65-4222-b6d0-0407b98b9b34" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:18:18.994680 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994646    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsm77\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-kube-api-access-wsm77\") pod \"86547d63-5a65-4222-b6d0-0407b98b9b34\" (UID: \"86547d63-5a65-4222-b6d0-0407b98b9b34\") "
May 25 11:18:18.994996 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994884    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86547d63-5a65-4222-b6d0-0407b98b9b34-ca-trust-extracted\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:18:18.994996 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.994934    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "86547d63-5a65-4222-b6d0-0407b98b9b34" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:18:18.995295 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.995259    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "86547d63-5a65-4222-b6d0-0407b98b9b34" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:18:18.996817 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.996795    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "86547d63-5a65-4222-b6d0-0407b98b9b34" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:18:18.997112 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.997084    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "86547d63-5a65-4222-b6d0-0407b98b9b34" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:18:18.997112 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.997087    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "86547d63-5a65-4222-b6d0-0407b98b9b34" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:18:18.997257 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.997178    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "86547d63-5a65-4222-b6d0-0407b98b9b34" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:18:18.997315 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:18.997299    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-kube-api-access-wsm77" (OuterVolumeSpecName: "kube-api-access-wsm77") pod "86547d63-5a65-4222-b6d0-0407b98b9b34" (UID: "86547d63-5a65-4222-b6d0-0407b98b9b34"). InnerVolumeSpecName "kube-api-access-wsm77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:18:19.024590 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.024395    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlbp4\""
May 25 11:18:19.033141 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.032687    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gktpf"
May 25 11:18:19.045879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.045819    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kttpg\""
May 25 11:18:19.048842 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.048820    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlfq6"
May 25 11:18:19.095634 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.095508    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-trusted-ca\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:18:19.095634 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.095540    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-image-registry-private-configuration\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:18:19.095634 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.095557    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86547d63-5a65-4222-b6d0-0407b98b9b34-installation-pull-secrets\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:18:19.095634 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.095571    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:18:19.095634 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.095585    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86547d63-5a65-4222-b6d0-0407b98b9b34-registry-certificates\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:18:19.095634 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.095599    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-bound-sa-token\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:18:19.095634 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.095613    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsm77\" (UniqueName: \"kubernetes.io/projected/86547d63-5a65-4222-b6d0-0407b98b9b34-kube-api-access-wsm77\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:18:19.163567 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.163508    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qqxf"]
May 25 11:18:19.170815 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:18:19.170740    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9135fe8_f84c_4c3d_869c_2f4c2a5e49bc.slice/crio-65a0222e55664a6689379170cf51c6a253103d5b2b0009ee07c90e43804ec527 WatchSource:0}: Error finding container 65a0222e55664a6689379170cf51c6a253103d5b2b0009ee07c90e43804ec527: Status 404 returned error can't find the container with id 65a0222e55664a6689379170cf51c6a253103d5b2b0009ee07c90e43804ec527
May 25 11:18:19.229197 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.229170    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlfq6"]
May 25 11:18:19.230671 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.230627    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gktpf"]
May 25 11:18:19.232131 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:18:19.232097    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bbe64f_7412_4907_bd9f_658a5034f66c.slice/crio-69f47496430da194af57d72992adbb2c8366e90b2056ccbab66e7df4def177a8 WatchSource:0}: Error finding container 69f47496430da194af57d72992adbb2c8366e90b2056ccbab66e7df4def177a8: Status 404 returned error can't find the container with id 69f47496430da194af57d72992adbb2c8366e90b2056ccbab66e7df4def177a8
May 25 11:18:19.233171 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:18:19.233114    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf5f84b_f809_49d2_8c90_bdedd5b862e8.slice/crio-233c40a73894d8a2a0170ba27b51ae0328c655824910b0c14e0e6b6ed1bcfaeb WatchSource:0}: Error finding container 233c40a73894d8a2a0170ba27b51ae0328c655824910b0c14e0e6b6ed1bcfaeb: Status 404 returned error can't find the container with id 233c40a73894d8a2a0170ba27b51ae0328c655824910b0c14e0e6b6ed1bcfaeb
May 25 11:18:19.883218 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.883187    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlfq6" event={"ID":"f2bbe64f-7412-4907-bd9f-658a5034f66c","Type":"ContainerStarted","Data":"69f47496430da194af57d72992adbb2c8366e90b2056ccbab66e7df4def177a8"}
May 25 11:18:19.884425 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.884379    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gktpf" event={"ID":"ebf5f84b-f809-49d2-8c90-bdedd5b862e8","Type":"ContainerStarted","Data":"233c40a73894d8a2a0170ba27b51ae0328c655824910b0c14e0e6b6ed1bcfaeb"}
May 25 11:18:19.886979 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.886928    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qqxf" event={"ID":"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc","Type":"ContainerStarted","Data":"06d96228017e6688a0f343eb278753e555d6be894687dffd0aaa497a67b3af4d"}
May 25 11:18:19.886979 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.886963    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qqxf" event={"ID":"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc","Type":"ContainerStarted","Data":"65a0222e55664a6689379170cf51c6a253103d5b2b0009ee07c90e43804ec527"}
May 25 11:18:19.888954 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.888927    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2l9j" event={"ID":"c8d62b8b-fa29-494d-a37a-3436347370ed","Type":"ContainerStarted","Data":"88de35b2b3359398e0753454e3f65dd3a27d5afe3dbd9626233ad0830e154953"}
May 25 11:18:19.890113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.890048    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59fc58bc-55k7g"
May 25 11:18:19.892467 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.892106    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp" event={"ID":"91347ebc-aca7-4076-a5d2-487b1f433bb1","Type":"ContainerStarted","Data":"bd2256809437dfd178508d103cfb508b4454085e9f70a9c9bc6a05216d3bc9b3"}
May 25 11:18:19.951910 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.951853    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d59fc58bc-55k7g"]
May 25 11:18:19.969984 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:19.969921    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d59fc58bc-55k7g"]
May 25 11:18:20.570037 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:20.569677    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86547d63-5a65-4222-b6d0-0407b98b9b34" path="/var/lib/kubelet/pods/86547d63-5a65-4222-b6d0-0407b98b9b34/volumes"
May 25 11:18:20.896425 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:20.896304    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qqxf" event={"ID":"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc","Type":"ContainerStarted","Data":"a5b8e0153db16fbfda4c5930fecd84a8a8c1ac6872a7215b6d56a3fc74d938a8"}
May 25 11:18:20.899753 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:20.898937    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2l9j" event={"ID":"c8d62b8b-fa29-494d-a37a-3436347370ed","Type":"ContainerStarted","Data":"db76d8f45cc6d9ca44617d0a5aa7beff4dba38714213a6f733fea59a643b06e2"}
May 25 11:18:20.930710 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:20.930445    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z2l9j" podStartSLOduration=67.968835569 podStartE2EDuration="1m8.930427061s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:18:18.781500893 +0000 UTC m=+66.777675699" lastFinishedPulling="2026-05-25 11:18:19.743092386 +0000 UTC m=+67.739267191" observedRunningTime="2026-05-25 11:18:20.927277172 +0000 UTC m=+68.923451984" watchObservedRunningTime="2026-05-25 11:18:20.930427061 +0000 UTC m=+68.926601876"
May 25 11:18:24.912106 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.912019    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gzg8z" event={"ID":"607c3221-179d-44be-81d0-930ca62118ad","Type":"ContainerStarted","Data":"8772684d7fe3d5061bfb77737efdacd6ed087208c1a8020150ca6977d56ab579"}
May 25 11:18:24.912562 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.912145    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:18:24.913370 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.913344    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp" event={"ID":"91347ebc-aca7-4076-a5d2-487b1f433bb1","Type":"ContainerStarted","Data":"7be9fe66e3b32c492d36327ac06df190f264e76cf5a0d4c62b1c0a31ac8c071d"}
May 25 11:18:24.914568 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.914547    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlfq6" event={"ID":"f2bbe64f-7412-4907-bd9f-658a5034f66c","Type":"ContainerStarted","Data":"8e7ea12abfd1cd509e9cd6ee2cca08f29208f72131e2c406ddba06c8caa499bd"}
May 25 11:18:24.916053 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.916031    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gktpf" event={"ID":"ebf5f84b-f809-49d2-8c90-bdedd5b862e8","Type":"ContainerStarted","Data":"81a35043aa30f4064df12441539e7152246fb9b9c5d2fb95bf59a5bea5e744ed"}
May 25 11:18:24.916150 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.916058    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gktpf" event={"ID":"ebf5f84b-f809-49d2-8c90-bdedd5b862e8","Type":"ContainerStarted","Data":"5de47020bc89bbcd33edcb2e088e3c855f3c7cce33eba6776a9f54adafcc2b8c"}
May 25 11:18:24.916215 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.916154    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gktpf"
May 25 11:18:24.917726 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.917707    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qqxf" event={"ID":"b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc","Type":"ContainerStarted","Data":"5da7cf2b0fe23b8f34222fab63d179862a56ecf0f762135a9b9cb0cd5bc5dce7"}
May 25 11:18:24.931153 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.931118    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gzg8z" podStartSLOduration=67.059926787 podStartE2EDuration="1m12.931109206s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:18:18.684087817 +0000 UTC m=+66.680262621" lastFinishedPulling="2026-05-25 11:18:24.555270241 +0000 UTC m=+72.551445040" observedRunningTime="2026-05-25 11:18:24.930563372 +0000 UTC m=+72.926738208" watchObservedRunningTime="2026-05-25 11:18:24.931109206 +0000 UTC m=+72.927284039"
May 25 11:18:24.946623 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.946585    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vlfq6" podStartSLOduration=33.620393241 podStartE2EDuration="38.946574527s" podCreationTimestamp="2026-05-25 11:17:46 +0000 UTC" firstStartedPulling="2026-05-25 11:18:19.235271665 +0000 UTC m=+67.231446456" lastFinishedPulling="2026-05-25 11:18:24.561452949 +0000 UTC m=+72.557627742" observedRunningTime="2026-05-25 11:18:24.946238054 +0000 UTC m=+72.942412877" watchObservedRunningTime="2026-05-25 11:18:24.946574527 +0000 UTC m=+72.942749339"
May 25 11:18:24.967063 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.967024    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7qqxf" podStartSLOduration=1.649829033 podStartE2EDuration="6.966990569s" podCreationTimestamp="2026-05-25 11:18:18 +0000 UTC" firstStartedPulling="2026-05-25 11:18:19.239404022 +0000 UTC m=+67.235578827" lastFinishedPulling="2026-05-25 11:18:24.556565563 +0000 UTC m=+72.552740363" observedRunningTime="2026-05-25 11:18:24.96601137 +0000 UTC m=+72.962186184" watchObservedRunningTime="2026-05-25 11:18:24.966990569 +0000 UTC m=+72.963165382"
May 25 11:18:24.984438 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:24.984393    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-gktpf" podStartSLOduration=33.665360693 podStartE2EDuration="38.984372655s" podCreationTimestamp="2026-05-25 11:17:46 +0000 UTC" firstStartedPulling="2026-05-25 11:18:19.236385368 +0000 UTC m=+67.232560165" lastFinishedPulling="2026-05-25 11:18:24.555397336 +0000 UTC m=+72.551572127" observedRunningTime="2026-05-25 11:18:24.983201428 +0000 UTC m=+72.979376240" watchObservedRunningTime="2026-05-25 11:18:24.984372655 +0000 UTC m=+72.980547469"
May 25 11:18:26.537638 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.536027    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"]
May 25 11:18:26.543444 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.543421    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.547351 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.547327    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 11:18:26.547351 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.547327    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 25 11:18:26.553392 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.553330    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 11:18:26.553903 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.553882    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 25 11:18:26.554445 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.554431    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dl29g\""
May 25 11:18:26.565530 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.565502    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 11:18:26.573598 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.573575    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"]
May 25 11:18:26.651498 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.651457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s549p\" (UniqueName: \"kubernetes.io/projected/285638e3-f856-4909-9a8a-37404fc1617f-kube-api-access-s549p\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.651677 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.651515    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/285638e3-f856-4909-9a8a-37404fc1617f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.651677 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.651537    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/285638e3-f856-4909-9a8a-37404fc1617f-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.651677 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.651565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/285638e3-f856-4909-9a8a-37404fc1617f-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.752881 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.752839    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/285638e3-f856-4909-9a8a-37404fc1617f-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.753069 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.752898    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s549p\" (UniqueName: \"kubernetes.io/projected/285638e3-f856-4909-9a8a-37404fc1617f-kube-api-access-s549p\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.753069 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.752962    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/285638e3-f856-4909-9a8a-37404fc1617f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.753069 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.752995    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/285638e3-f856-4909-9a8a-37404fc1617f-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.753069 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:26.753003    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 25 11:18:26.753251 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:18:26.753080    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285638e3-f856-4909-9a8a-37404fc1617f-prometheus-operator-tls podName:285638e3-f856-4909-9a8a-37404fc1617f nodeName:}" failed. No retries permitted until 2026-05-25 11:18:27.253059045 +0000 UTC m=+75.249233854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/285638e3-f856-4909-9a8a-37404fc1617f-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-9hphj" (UID: "285638e3-f856-4909-9a8a-37404fc1617f") : secret "prometheus-operator-tls" not found
May 25 11:18:26.753703 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.753681    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/285638e3-f856-4909-9a8a-37404fc1617f-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.755582 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.755559    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/285638e3-f856-4909-9a8a-37404fc1617f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:26.763571 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:26.763545    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s549p\" (UniqueName: \"kubernetes.io/projected/285638e3-f856-4909-9a8a-37404fc1617f-kube-api-access-s549p\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:27.257229 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:27.257203    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/285638e3-f856-4909-9a8a-37404fc1617f-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:27.259292 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:27.259267    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/285638e3-f856-4909-9a8a-37404fc1617f-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-9hphj\" (UID: \"285638e3-f856-4909-9a8a-37404fc1617f\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:27.454031 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:27.453999    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"
May 25 11:18:27.572959 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:27.572929    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-9hphj"]
May 25 11:18:27.576219 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:18:27.576194    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285638e3_f856_4909_9a8a_37404fc1617f.slice/crio-9dc9459575ced5585375865c37dac306f530dde149a4a1b4801b87247ca344d8 WatchSource:0}: Error finding container 9dc9459575ced5585375865c37dac306f530dde149a4a1b4801b87247ca344d8: Status 404 returned error can't find the container with id 9dc9459575ced5585375865c37dac306f530dde149a4a1b4801b87247ca344d8
May 25 11:18:27.928102 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:27.928010    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp" event={"ID":"91347ebc-aca7-4076-a5d2-487b1f433bb1","Type":"ContainerStarted","Data":"d5e78d966cef1f91169151206c83c5caf2f9ea600de950795b3530ff8deeeaf5"}
May 25 11:18:27.928102 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:27.928052    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp" event={"ID":"91347ebc-aca7-4076-a5d2-487b1f433bb1","Type":"ContainerStarted","Data":"b46c61de15f3f05843db3f4ec6147e1dc06a00f82cc8d2da9ac1d194b7a03508"}
May 25 11:18:27.929047 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:27.929024    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj" event={"ID":"285638e3-f856-4909-9a8a-37404fc1617f","Type":"ContainerStarted","Data":"9dc9459575ced5585375865c37dac306f530dde149a4a1b4801b87247ca344d8"}
May 25 11:18:27.951135 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:27.950855    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54647fd587-wwkzp" podStartSLOduration=1.651525905 podStartE2EDuration="9.95083939s" podCreationTimestamp="2026-05-25 11:18:18 +0000 UTC" firstStartedPulling="2026-05-25 11:18:18.932629451 +0000 UTC m=+66.928804241" lastFinishedPulling="2026-05-25 11:18:27.231942936 +0000 UTC m=+75.228117726" observedRunningTime="2026-05-25 11:18:27.949150405 +0000 UTC m=+75.945325219" watchObservedRunningTime="2026-05-25 11:18:27.95083939 +0000 UTC m=+75.947014203"
May 25 11:18:29.936361 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:29.936327    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj" event={"ID":"285638e3-f856-4909-9a8a-37404fc1617f","Type":"ContainerStarted","Data":"2bab7b9bed087e9833d99794a4fa05c5d9ed94a9782d8a1b730a5a2b0aab3468"}
May 25 11:18:29.936744 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:29.936370    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj" event={"ID":"285638e3-f856-4909-9a8a-37404fc1617f","Type":"ContainerStarted","Data":"907af43e101c51a024ea8fcfb21b82dcc94317131b9325c78ca97e344b5beaa5"}
May 25 11:18:29.960430 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:29.960386    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-9hphj" podStartSLOduration=2.477848522 podStartE2EDuration="3.96037482s" podCreationTimestamp="2026-05-25 11:18:26 +0000 UTC" firstStartedPulling="2026-05-25 11:18:27.577906742 +0000 UTC m=+75.574081533" lastFinishedPulling="2026-05-25 11:18:29.06043304 +0000 UTC m=+77.056607831" observedRunningTime="2026-05-25 11:18:29.956193735 +0000 UTC m=+77.952368548" watchObservedRunningTime="2026-05-25 11:18:29.96037482 +0000 UTC m=+77.956549632"
May 25 11:18:32.215494 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.215453    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pwkgz"]
May 25 11:18:32.217485 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.217463    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.224650 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.224625    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 11:18:32.224751 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.224698    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 11:18:32.224838 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.224824    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wmnhm\""
May 25 11:18:32.225900 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.225884    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 11:18:32.291827 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.291790    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/418654e3-284e-4516-8934-67302c832059-node-exporter-wtmp\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.291827 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.291832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/418654e3-284e-4516-8934-67302c832059-node-exporter-tls\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.292022 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.291851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/418654e3-284e-4516-8934-67302c832059-metrics-client-ca\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.292022 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.291866    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2z6\" (UniqueName: \"kubernetes.io/projected/418654e3-284e-4516-8934-67302c832059-kube-api-access-hs2z6\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.292022 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.291886    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/418654e3-284e-4516-8934-67302c832059-sys\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.292022 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.291937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/418654e3-284e-4516-8934-67302c832059-root\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.292022 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.291986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/418654e3-284e-4516-8934-67302c832059-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.292180 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.292033    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/418654e3-284e-4516-8934-67302c832059-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.292180 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.292056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/418654e3-284e-4516-8934-67302c832059-node-exporter-textfile\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393136 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393105    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/418654e3-284e-4516-8934-67302c832059-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393278 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393141    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/418654e3-284e-4516-8934-67302c832059-node-exporter-textfile\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393278 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393167    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/418654e3-284e-4516-8934-67302c832059-node-exporter-wtmp\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393278 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393185    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/418654e3-284e-4516-8934-67302c832059-node-exporter-tls\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393278 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393216    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/418654e3-284e-4516-8934-67302c832059-metrics-client-ca\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393278 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393238    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2z6\" (UniqueName: \"kubernetes.io/projected/418654e3-284e-4516-8934-67302c832059-kube-api-access-hs2z6\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393278 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393264    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/418654e3-284e-4516-8934-67302c832059-sys\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393307    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/418654e3-284e-4516-8934-67302c832059-sys\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393323    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/418654e3-284e-4516-8934-67302c832059-node-exporter-wtmp\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393504    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/418654e3-284e-4516-8934-67302c832059-root\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393537    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/418654e3-284e-4516-8934-67302c832059-node-exporter-textfile\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393544    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/418654e3-284e-4516-8934-67302c832059-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393595    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/418654e3-284e-4516-8934-67302c832059-root\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.393850 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.393820    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/418654e3-284e-4516-8934-67302c832059-metrics-client-ca\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.394017 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.394000    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/418654e3-284e-4516-8934-67302c832059-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.396008 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.395983    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/418654e3-284e-4516-8934-67302c832059-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.396162 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.396140    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/418654e3-284e-4516-8934-67302c832059-node-exporter-tls\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.417487 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.417467    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2z6\" (UniqueName: \"kubernetes.io/projected/418654e3-284e-4516-8934-67302c832059-kube-api-access-hs2z6\") pod \"node-exporter-pwkgz\" (UID: \"418654e3-284e-4516-8934-67302c832059\") " pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.526305 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.526277    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwkgz"
May 25 11:18:32.533787 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:18:32.533749    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418654e3_284e_4516_8934_67302c832059.slice/crio-0dd798faac3bef8451a8006758b294d9fcb30a33960ad0297989ba2de8b985f6 WatchSource:0}: Error finding container 0dd798faac3bef8451a8006758b294d9fcb30a33960ad0297989ba2de8b985f6: Status 404 returned error can't find the container with id 0dd798faac3bef8451a8006758b294d9fcb30a33960ad0297989ba2de8b985f6
May 25 11:18:32.946177 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:32.946082    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwkgz" event={"ID":"418654e3-284e-4516-8934-67302c832059","Type":"ContainerStarted","Data":"0dd798faac3bef8451a8006758b294d9fcb30a33960ad0297989ba2de8b985f6"}
May 25 11:18:33.952057 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:33.952018    2569 generic.go:358] "Generic (PLEG): container finished" podID="418654e3-284e-4516-8934-67302c832059" containerID="701c34281d700146a9f696e4fcee53359f175c6c960d0a6dd00d361f0060e646" exitCode=0
May 25 11:18:33.952496 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:33.952070    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwkgz" event={"ID":"418654e3-284e-4516-8934-67302c832059","Type":"ContainerDied","Data":"701c34281d700146a9f696e4fcee53359f175c6c960d0a6dd00d361f0060e646"}
May 25 11:18:34.924262 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:34.924231    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gktpf"
May 25 11:18:34.956709 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:34.956671    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwkgz" event={"ID":"418654e3-284e-4516-8934-67302c832059","Type":"ContainerStarted","Data":"5c41aeade31938290193f2238d458af02d122b52beb62e8c7455686703ea638b"}
May 25 11:18:34.956709 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:34.956712    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwkgz" event={"ID":"418654e3-284e-4516-8934-67302c832059","Type":"ContainerStarted","Data":"0fa0412ce043771de69b232fe4afa448ace27d72083b2f69bb4f4d3a9fa019c7"}
May 25 11:18:34.978876 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:34.978822    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pwkgz" podStartSLOduration=2.1435657519999998 podStartE2EDuration="2.97880723s" podCreationTimestamp="2026-05-25 11:18:32 +0000 UTC" firstStartedPulling="2026-05-25 11:18:32.535390565 +0000 UTC m=+80.531565369" lastFinishedPulling="2026-05-25 11:18:33.37063204 +0000 UTC m=+81.366806847" observedRunningTime="2026-05-25 11:18:34.977697557 +0000 UTC m=+82.973872384" watchObservedRunningTime="2026-05-25 11:18:34.97880723 +0000 UTC m=+82.974982042"
May 25 11:18:35.449321 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.449285    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-549f959ff5-8kwk2"]
May 25 11:18:35.452299 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.452284    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.459526 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.459507    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 11:18:35.465258 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.465240    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 11:18:35.465333 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.465292    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 11:18:35.465386 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.465335    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 11:18:35.465386 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.465335    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 11:18:35.465547 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.465535    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 11:18:35.468233 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.468218    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wkb8g\""
May 25 11:18:35.479026 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.478999    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 11:18:35.488471 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.488452    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549f959ff5-8kwk2"]
May 25 11:18:35.499463 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.499442    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 11:18:35.522493 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.522471    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-serving-cert\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.522566 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.522505    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-trusted-ca-bundle\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.522566 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.522528    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-console-config\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.522566 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.522547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-oauth-serving-cert\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.522659 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.522609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-service-ca\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.522659 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.522638    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-oauth-config\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.522659 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.522656    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrcw\" (UniqueName: \"kubernetes.io/projected/0ac7d659-26a1-4988-a187-8c01f1afc450-kube-api-access-jgrcw\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.623869 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.623840    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-serving-cert\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.623969 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.623886    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-trusted-ca-bundle\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.623969 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.623932    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-console-config\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.623969 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.623959    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-oauth-serving-cert\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.624077 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.623988    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-service-ca\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.624077 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.624016    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-oauth-config\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.624077 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.624040    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrcw\" (UniqueName: \"kubernetes.io/projected/0ac7d659-26a1-4988-a187-8c01f1afc450-kube-api-access-jgrcw\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.624687 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.624661    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-console-config\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.624806 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.624744    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-service-ca\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.624910 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.624891    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-trusted-ca-bundle\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.624972 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.624924    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-oauth-serving-cert\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.626444 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.626427    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-serving-cert\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.626565 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.626546    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-oauth-config\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.646285 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.646264    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrcw\" (UniqueName: \"kubernetes.io/projected/0ac7d659-26a1-4988-a187-8c01f1afc450-kube-api-access-jgrcw\") pod \"console-549f959ff5-8kwk2\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") " pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.761355 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.761321    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:35.879473 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.879450    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549f959ff5-8kwk2"]
May 25 11:18:35.881896 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:18:35.881873    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac7d659_26a1_4988_a187_8c01f1afc450.slice/crio-930ffa5874584e8db2cb9217dc9b251ec770617e7f0199f39b746dba44e6566b WatchSource:0}: Error finding container 930ffa5874584e8db2cb9217dc9b251ec770617e7f0199f39b746dba44e6566b: Status 404 returned error can't find the container with id 930ffa5874584e8db2cb9217dc9b251ec770617e7f0199f39b746dba44e6566b
May 25 11:18:35.963491 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:35.963458    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549f959ff5-8kwk2" event={"ID":"0ac7d659-26a1-4988-a187-8c01f1afc450","Type":"ContainerStarted","Data":"930ffa5874584e8db2cb9217dc9b251ec770617e7f0199f39b746dba44e6566b"}
May 25 11:18:38.973695 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:38.973611    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549f959ff5-8kwk2" event={"ID":"0ac7d659-26a1-4988-a187-8c01f1afc450","Type":"ContainerStarted","Data":"620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5"}
May 25 11:18:38.993297 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:38.993255    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-549f959ff5-8kwk2" podStartSLOduration=1.367250887 podStartE2EDuration="3.993239969s" podCreationTimestamp="2026-05-25 11:18:35 +0000 UTC" firstStartedPulling="2026-05-25 11:18:35.883735418 +0000 UTC m=+83.879910210" lastFinishedPulling="2026-05-25 11:18:38.509724488 +0000 UTC m=+86.505899292" observedRunningTime="2026-05-25 11:18:38.992825286 +0000 UTC m=+86.989000112" watchObservedRunningTime="2026-05-25 11:18:38.993239969 +0000 UTC m=+86.989414783"
May 25 11:18:41.722171 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:41.722141    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549f959ff5-8kwk2"]
May 25 11:18:45.761856 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:45.761818    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:18:55.923281 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:18:55.923246    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gzg8z"
May 25 11:19:06.741787 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:06.741709    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-549f959ff5-8kwk2" podUID="0ac7d659-26a1-4988-a187-8c01f1afc450" containerName="console" containerID="cri-o://620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5" gracePeriod=15
May 25 11:19:06.977482 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:06.977461    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-549f959ff5-8kwk2_0ac7d659-26a1-4988-a187-8c01f1afc450/console/0.log"
May 25 11:19:06.977592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:06.977523    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:19:07.047592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.047527    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-549f959ff5-8kwk2_0ac7d659-26a1-4988-a187-8c01f1afc450/console/0.log"
May 25 11:19:07.047592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.047568    2569 generic.go:358] "Generic (PLEG): container finished" podID="0ac7d659-26a1-4988-a187-8c01f1afc450" containerID="620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5" exitCode=2
May 25 11:19:07.047836 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.047631    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549f959ff5-8kwk2" event={"ID":"0ac7d659-26a1-4988-a187-8c01f1afc450","Type":"ContainerDied","Data":"620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5"}
May 25 11:19:07.047836 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.047638    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549f959ff5-8kwk2"
May 25 11:19:07.047836 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.047665    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549f959ff5-8kwk2" event={"ID":"0ac7d659-26a1-4988-a187-8c01f1afc450","Type":"ContainerDied","Data":"930ffa5874584e8db2cb9217dc9b251ec770617e7f0199f39b746dba44e6566b"}
May 25 11:19:07.047836 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.047686    2569 scope.go:117] "RemoveContainer" containerID="620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5"
May 25 11:19:07.048429 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.048411    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-serving-cert\") pod \"0ac7d659-26a1-4988-a187-8c01f1afc450\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") "
May 25 11:19:07.048478 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.048438    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-oauth-config\") pod \"0ac7d659-26a1-4988-a187-8c01f1afc450\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") "
May 25 11:19:07.048478 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.048458    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgrcw\" (UniqueName: \"kubernetes.io/projected/0ac7d659-26a1-4988-a187-8c01f1afc450-kube-api-access-jgrcw\") pod \"0ac7d659-26a1-4988-a187-8c01f1afc450\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") "
May 25 11:19:07.048571 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.048485    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-oauth-serving-cert\") pod \"0ac7d659-26a1-4988-a187-8c01f1afc450\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") "
May 25 11:19:07.048571 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.048551    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-console-config\") pod \"0ac7d659-26a1-4988-a187-8c01f1afc450\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") "
May 25 11:19:07.048665 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.048586    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-trusted-ca-bundle\") pod \"0ac7d659-26a1-4988-a187-8c01f1afc450\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") "
May 25 11:19:07.048665 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.048613    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-service-ca\") pod \"0ac7d659-26a1-4988-a187-8c01f1afc450\" (UID: \"0ac7d659-26a1-4988-a187-8c01f1afc450\") "
May 25 11:19:07.049299 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.049004    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-console-config" (OuterVolumeSpecName: "console-config") pod "0ac7d659-26a1-4988-a187-8c01f1afc450" (UID: "0ac7d659-26a1-4988-a187-8c01f1afc450"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:07.049299 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.049009    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0ac7d659-26a1-4988-a187-8c01f1afc450" (UID: "0ac7d659-26a1-4988-a187-8c01f1afc450"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:07.049299 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.049060    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0ac7d659-26a1-4988-a187-8c01f1afc450" (UID: "0ac7d659-26a1-4988-a187-8c01f1afc450"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:07.049299 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.049239    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-trusted-ca-bundle\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:19:07.049299 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.049259    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-oauth-serving-cert\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:19:07.049299 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.049276    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-console-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:19:07.049299 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.049265    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-service-ca" (OuterVolumeSpecName: "service-ca") pod "0ac7d659-26a1-4988-a187-8c01f1afc450" (UID: "0ac7d659-26a1-4988-a187-8c01f1afc450"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:07.050953 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.050922    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac7d659-26a1-4988-a187-8c01f1afc450-kube-api-access-jgrcw" (OuterVolumeSpecName: "kube-api-access-jgrcw") pod "0ac7d659-26a1-4988-a187-8c01f1afc450" (UID: "0ac7d659-26a1-4988-a187-8c01f1afc450"). InnerVolumeSpecName "kube-api-access-jgrcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:07.051057 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.050951    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0ac7d659-26a1-4988-a187-8c01f1afc450" (UID: "0ac7d659-26a1-4988-a187-8c01f1afc450"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:07.051057 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.051038    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0ac7d659-26a1-4988-a187-8c01f1afc450" (UID: "0ac7d659-26a1-4988-a187-8c01f1afc450"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:07.062707 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.062691    2569 scope.go:117] "RemoveContainer" containerID="620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5"
May 25 11:19:07.063005 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:19:07.062975    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5\": container with ID starting with 620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5 not found: ID does not exist" containerID="620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5"
May 25 11:19:07.063083 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.063015    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5"} err="failed to get container status \"620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5\": rpc error: code = NotFound desc = could not find container \"620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5\": container with ID starting with 620dedf09f14651835efdc0dfca03f74be4cd4b93dc40df826e11ed8c82a4ea5 not found: ID does not exist"
May 25 11:19:07.149536 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.149507    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-serving-cert\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:19:07.149536 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.149535    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ac7d659-26a1-4988-a187-8c01f1afc450-console-oauth-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:19:07.149664 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.149545    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrcw\" (UniqueName: \"kubernetes.io/projected/0ac7d659-26a1-4988-a187-8c01f1afc450-kube-api-access-jgrcw\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:19:07.149664 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.149556    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7d659-26a1-4988-a187-8c01f1afc450-service-ca\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:19:07.369016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.368987    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549f959ff5-8kwk2"]
May 25 11:19:07.372902 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:07.372880    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-549f959ff5-8kwk2"]
May 25 11:19:08.569200 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:08.569164    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ac7d659-26a1-4988-a187-8c01f1afc450" path="/var/lib/kubelet/pods/0ac7d659-26a1-4988-a187-8c01f1afc450/volumes"
May 25 11:19:59.453960 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.453863    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dbffcd589-spwxn"]
May 25 11:19:59.454503 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.454113    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac7d659-26a1-4988-a187-8c01f1afc450" containerName="console"
May 25 11:19:59.454503 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.454125    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac7d659-26a1-4988-a187-8c01f1afc450" containerName="console"
May 25 11:19:59.454503 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.454174    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="0ac7d659-26a1-4988-a187-8c01f1afc450" containerName="console"
May 25 11:19:59.456837 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.456815    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.460036 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.460012    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 11:19:59.460141 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.460020    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 11:19:59.461036 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.461015    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 11:19:59.461170 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.461065    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 11:19:59.461170 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.461124    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 11:19:59.461170 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.461164    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 11:19:59.461170 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.461173    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 11:19:59.461596 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.461581    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wkb8g\""
May 25 11:19:59.465723 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.465702    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 11:19:59.469307 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.469285    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dbffcd589-spwxn"]
May 25 11:19:59.594040 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.594003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-oauth-config\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.594040 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.594043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-service-ca\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.594279 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.594069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-serving-cert\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.594279 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.594112    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-config\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.594279 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.594138    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-trusted-ca-bundle\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.594279 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.594214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-oauth-serving-cert\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.594279 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.594265    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m675v\" (UniqueName: \"kubernetes.io/projected/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-kube-api-access-m675v\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.695376 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.695339    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-serving-cert\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.695564 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.695387    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-config\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.695564 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.695426    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-trusted-ca-bundle\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.695564 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.695457    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-oauth-serving-cert\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.695564 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.695484    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m675v\" (UniqueName: \"kubernetes.io/projected/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-kube-api-access-m675v\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.695564 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.695510    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-oauth-config\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.695564 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.695545    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-service-ca\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.696330 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.696283    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-config\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.696459 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.696368    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-service-ca\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.696459 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.696399    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-oauth-serving-cert\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.696459 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.696401    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-trusted-ca-bundle\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.698240 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.698211    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-oauth-config\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.699187 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.699165    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-serving-cert\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.711734 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.711682    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m675v\" (UniqueName: \"kubernetes.io/projected/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-kube-api-access-m675v\") pod \"console-5dbffcd589-spwxn\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") " pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.766845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.766808    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:19:59.888659 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:19:59.888634    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dbffcd589-spwxn"]
May 25 11:19:59.890483 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:19:59.890458    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101c50c2_7f93_4b5d_a18f_a1041de9fb2d.slice/crio-5419448fdc4c097d094303121d4170ca4847aa7ae50b451d12077b5a4f6a6ada WatchSource:0}: Error finding container 5419448fdc4c097d094303121d4170ca4847aa7ae50b451d12077b5a4f6a6ada: Status 404 returned error can't find the container with id 5419448fdc4c097d094303121d4170ca4847aa7ae50b451d12077b5a4f6a6ada
May 25 11:20:00.185140 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:20:00.185097    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbffcd589-spwxn" event={"ID":"101c50c2-7f93-4b5d-a18f-a1041de9fb2d","Type":"ContainerStarted","Data":"752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c"}
May 25 11:20:00.185140 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:20:00.185143    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbffcd589-spwxn" event={"ID":"101c50c2-7f93-4b5d-a18f-a1041de9fb2d","Type":"ContainerStarted","Data":"5419448fdc4c097d094303121d4170ca4847aa7ae50b451d12077b5a4f6a6ada"}
May 25 11:20:00.203476 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:20:00.203424    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5dbffcd589-spwxn" podStartSLOduration=1.203410069 podStartE2EDuration="1.203410069s" podCreationTimestamp="2026-05-25 11:19:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:20:00.202378326 +0000 UTC m=+168.198553131" watchObservedRunningTime="2026-05-25 11:20:00.203410069 +0000 UTC m=+168.199584882"
May 25 11:20:09.767207 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:20:09.767171    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:20:09.767207 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:20:09.767217    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:20:09.772313 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:20:09.772290    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:20:10.213507 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:20:10.213428    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:21:18.971120 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:18.971087    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-58cd66696f-f8fld"]
May 25 11:21:18.974031 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:18.974014    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:18.986122 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:18.986096    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cd66696f-f8fld"]
May 25 11:21:19.050104 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.050076    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-service-ca\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.050251 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.050110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-trusted-ca-bundle\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.050251 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.050135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-oauth-config\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.050251 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.050188    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-config\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.050251 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.050209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-serving-cert\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.050383 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.050291    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzv25\" (UniqueName: \"kubernetes.io/projected/f8bc6690-7d73-4195-9d0a-c25687df97b0-kube-api-access-hzv25\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.050383 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.050324    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-oauth-serving-cert\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.151000 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.150971    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hzv25\" (UniqueName: \"kubernetes.io/projected/f8bc6690-7d73-4195-9d0a-c25687df97b0-kube-api-access-hzv25\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.151165 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.151012    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-oauth-serving-cert\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.151239 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.151196    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-service-ca\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.151239 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.151228    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-trusted-ca-bundle\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.151347 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.151249    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-oauth-config\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.151347 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.151266    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-config\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.151443 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.151372    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-serving-cert\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.151690 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.151669    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-oauth-serving-cert\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.151940 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.151916    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-service-ca\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.152016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.151925    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-config\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.152074 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.152044    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-trusted-ca-bundle\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.153613 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.153590    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-oauth-config\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.153716 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.153699    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-serving-cert\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.158762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.158743    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzv25\" (UniqueName: \"kubernetes.io/projected/f8bc6690-7d73-4195-9d0a-c25687df97b0-kube-api-access-hzv25\") pod \"console-58cd66696f-f8fld\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") " pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.282487 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.282462    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:19.401724 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:19.401698    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cd66696f-f8fld"]
May 25 11:21:19.404465 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:21:19.404437    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bc6690_7d73_4195_9d0a_c25687df97b0.slice/crio-eda4bb2209d2ec2bfd619a4c1a0c47443542e76c359fa013e7ab36fc77fb5b4a WatchSource:0}: Error finding container eda4bb2209d2ec2bfd619a4c1a0c47443542e76c359fa013e7ab36fc77fb5b4a: Status 404 returned error can't find the container with id eda4bb2209d2ec2bfd619a4c1a0c47443542e76c359fa013e7ab36fc77fb5b4a
May 25 11:21:20.382926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:20.382891    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cd66696f-f8fld" event={"ID":"f8bc6690-7d73-4195-9d0a-c25687df97b0","Type":"ContainerStarted","Data":"05acf8796719a6c685579c4be99d5b221fe414328582edc5881b75c14f43a9fb"}
May 25 11:21:20.382926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:20.382927    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cd66696f-f8fld" event={"ID":"f8bc6690-7d73-4195-9d0a-c25687df97b0","Type":"ContainerStarted","Data":"eda4bb2209d2ec2bfd619a4c1a0c47443542e76c359fa013e7ab36fc77fb5b4a"}
May 25 11:21:20.404227 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:20.404184    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-58cd66696f-f8fld" podStartSLOduration=2.404171507 podStartE2EDuration="2.404171507s" podCreationTimestamp="2026-05-25 11:21:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:21:20.402206639 +0000 UTC m=+248.398381451" watchObservedRunningTime="2026-05-25 11:21:20.404171507 +0000 UTC m=+248.400346319"
May 25 11:21:29.282874 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:29.282742    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:29.282874 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:29.282830    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:29.287818 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:29.287797    2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:29.408192 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:29.408163    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:21:29.458085 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:29.458056    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dbffcd589-spwxn"]
May 25 11:21:54.476724 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.476686    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5dbffcd589-spwxn" podUID="101c50c2-7f93-4b5d-a18f-a1041de9fb2d" containerName="console" containerID="cri-o://752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c" gracePeriod=15
May 25 11:21:54.713438 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.713418    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5dbffcd589-spwxn_101c50c2-7f93-4b5d-a18f-a1041de9fb2d/console/0.log"
May 25 11:21:54.713547 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.713475    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:21:54.794879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.794853    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m675v\" (UniqueName: \"kubernetes.io/projected/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-kube-api-access-m675v\") pod \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") "
May 25 11:21:54.795067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.794895    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-service-ca\") pod \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") "
May 25 11:21:54.795067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.794929    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-oauth-serving-cert\") pod \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") "
May 25 11:21:54.795067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.794954    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-oauth-config\") pod \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") "
May 25 11:21:54.795067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.794970    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-serving-cert\") pod \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") "
May 25 11:21:54.795067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.794998    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-trusted-ca-bundle\") pod \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") "
May 25 11:21:54.795067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.795018    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-config\") pod \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\" (UID: \"101c50c2-7f93-4b5d-a18f-a1041de9fb2d\") "
May 25 11:21:54.795459 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.795427    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "101c50c2-7f93-4b5d-a18f-a1041de9fb2d" (UID: "101c50c2-7f93-4b5d-a18f-a1041de9fb2d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:54.795557 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.795463    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-service-ca" (OuterVolumeSpecName: "service-ca") pod "101c50c2-7f93-4b5d-a18f-a1041de9fb2d" (UID: "101c50c2-7f93-4b5d-a18f-a1041de9fb2d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:54.795557 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.795491    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "101c50c2-7f93-4b5d-a18f-a1041de9fb2d" (UID: "101c50c2-7f93-4b5d-a18f-a1041de9fb2d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:54.795665 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.795577    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-config" (OuterVolumeSpecName: "console-config") pod "101c50c2-7f93-4b5d-a18f-a1041de9fb2d" (UID: "101c50c2-7f93-4b5d-a18f-a1041de9fb2d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:54.796994 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.796966    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-kube-api-access-m675v" (OuterVolumeSpecName: "kube-api-access-m675v") pod "101c50c2-7f93-4b5d-a18f-a1041de9fb2d" (UID: "101c50c2-7f93-4b5d-a18f-a1041de9fb2d"). InnerVolumeSpecName "kube-api-access-m675v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:21:54.797098 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.797036    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "101c50c2-7f93-4b5d-a18f-a1041de9fb2d" (UID: "101c50c2-7f93-4b5d-a18f-a1041de9fb2d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:54.797098 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.797072    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "101c50c2-7f93-4b5d-a18f-a1041de9fb2d" (UID: "101c50c2-7f93-4b5d-a18f-a1041de9fb2d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:54.895493 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.895448    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-trusted-ca-bundle\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:21:54.895493 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.895491    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:21:54.895493 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.895501    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m675v\" (UniqueName: \"kubernetes.io/projected/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-kube-api-access-m675v\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:21:54.895694 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.895511    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-service-ca\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:21:54.895694 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.895519    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-oauth-serving-cert\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:21:54.895694 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.895531    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-oauth-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:21:54.895694 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:54.895544    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/101c50c2-7f93-4b5d-a18f-a1041de9fb2d-console-serving-cert\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:21:55.467681 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.467654    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5dbffcd589-spwxn_101c50c2-7f93-4b5d-a18f-a1041de9fb2d/console/0.log"
May 25 11:21:55.467926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.467696    2569 generic.go:358] "Generic (PLEG): container finished" podID="101c50c2-7f93-4b5d-a18f-a1041de9fb2d" containerID="752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c" exitCode=2
May 25 11:21:55.467926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.467782    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbffcd589-spwxn"
May 25 11:21:55.467926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.467797    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbffcd589-spwxn" event={"ID":"101c50c2-7f93-4b5d-a18f-a1041de9fb2d","Type":"ContainerDied","Data":"752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c"}
May 25 11:21:55.467926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.467832    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbffcd589-spwxn" event={"ID":"101c50c2-7f93-4b5d-a18f-a1041de9fb2d","Type":"ContainerDied","Data":"5419448fdc4c097d094303121d4170ca4847aa7ae50b451d12077b5a4f6a6ada"}
May 25 11:21:55.467926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.467851    2569 scope.go:117] "RemoveContainer" containerID="752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c"
May 25 11:21:55.475970 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.475796    2569 scope.go:117] "RemoveContainer" containerID="752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c"
May 25 11:21:55.476226 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:21:55.476203    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c\": container with ID starting with 752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c not found: ID does not exist" containerID="752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c"
May 25 11:21:55.476312 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.476237    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c"} err="failed to get container status \"752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c\": rpc error: code = NotFound desc = could not find container \"752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c\": container with ID starting with 752e1eccbcce9ef6d471644194be2cb7ec1f850709821e7ab0ba3cafc2a6b98c not found: ID does not exist"
May 25 11:21:55.490423 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.490404    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dbffcd589-spwxn"]
May 25 11:21:55.495844 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:55.495814    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dbffcd589-spwxn"]
May 25 11:21:56.568384 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:56.568351    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="101c50c2-7f93-4b5d-a18f-a1041de9fb2d" path="/var/lib/kubelet/pods/101c50c2-7f93-4b5d-a18f-a1041de9fb2d/volumes"
May 25 11:21:58.694615 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.694583    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"]
May 25 11:21:58.694993 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.694821    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101c50c2-7f93-4b5d-a18f-a1041de9fb2d" containerName="console"
May 25 11:21:58.694993 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.694832    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="101c50c2-7f93-4b5d-a18f-a1041de9fb2d" containerName="console"
May 25 11:21:58.694993 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.694882    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="101c50c2-7f93-4b5d-a18f-a1041de9fb2d" containerName="console"
May 25 11:21:58.699140 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.699122    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:58.702098 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.702075    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:21:58.702874 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.702847    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:21:58.702874 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.702847    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c4tqb\""
May 25 11:21:58.707218 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.707196    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"]
May 25 11:21:58.819806 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.819763    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7bmw\" (UniqueName: \"kubernetes.io/projected/f474090b-08ff-4e37-bb28-093556a57b03-kube-api-access-r7bmw\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:58.819954 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.819828    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:58.819954 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.819871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:58.921159 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.921129    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7bmw\" (UniqueName: \"kubernetes.io/projected/f474090b-08ff-4e37-bb28-093556a57b03-kube-api-access-r7bmw\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:58.921262 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.921170    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:58.921262 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.921188    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:58.921588 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.921570    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:58.921626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.921589    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:58.929942 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:58.929915    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7bmw\" (UniqueName: \"kubernetes.io/projected/f474090b-08ff-4e37-bb28-093556a57b03-kube-api-access-r7bmw\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:59.007967 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:59.007941    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:21:59.128530 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:59.128499    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"]
May 25 11:21:59.131416 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:21:59.131389    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf474090b_08ff_4e37_bb28_093556a57b03.slice/crio-e75ae2a82191e9adb0d1c0dfe3ecfd4b95d786bbfb188f59c88454c100c505c5 WatchSource:0}: Error finding container e75ae2a82191e9adb0d1c0dfe3ecfd4b95d786bbfb188f59c88454c100c505c5: Status 404 returned error can't find the container with id e75ae2a82191e9adb0d1c0dfe3ecfd4b95d786bbfb188f59c88454c100c505c5
May 25 11:21:59.477999 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:21:59.477920    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf" event={"ID":"f474090b-08ff-4e37-bb28-093556a57b03","Type":"ContainerStarted","Data":"e75ae2a82191e9adb0d1c0dfe3ecfd4b95d786bbfb188f59c88454c100c505c5"}
May 25 11:22:04.494728 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:04.494690    2569 generic.go:358] "Generic (PLEG): container finished" podID="f474090b-08ff-4e37-bb28-093556a57b03" containerID="682976f5b03edfdfb0e4010f5b1927e43eca69de8d1720f88df4754cb3483cd9" exitCode=0
May 25 11:22:04.495111 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:04.494734    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf" event={"ID":"f474090b-08ff-4e37-bb28-093556a57b03","Type":"ContainerDied","Data":"682976f5b03edfdfb0e4010f5b1927e43eca69de8d1720f88df4754cb3483cd9"}
May 25 11:22:07.505166 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:07.505130    2569 generic.go:358] "Generic (PLEG): container finished" podID="f474090b-08ff-4e37-bb28-093556a57b03" containerID="303adbcedade6c9f1c97547430c25f5f0a1e97a1e50e78c4b55ffe919af3889f" exitCode=0
May 25 11:22:07.505568 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:07.505188    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf" event={"ID":"f474090b-08ff-4e37-bb28-093556a57b03","Type":"ContainerDied","Data":"303adbcedade6c9f1c97547430c25f5f0a1e97a1e50e78c4b55ffe919af3889f"}
May 25 11:22:13.014752 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:13.014724    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:22:13.015228 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:13.014888    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:22:13.018717 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:13.018699    2569 kubelet.go:1628] "Image garbage collection succeeded"
May 25 11:22:13.523999 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:13.523963    2569 generic.go:358] "Generic (PLEG): container finished" podID="f474090b-08ff-4e37-bb28-093556a57b03" containerID="eed98312a4f9083fdebe34d291a53af8afe162df5587fc78d4043e599abf5efa" exitCode=0
May 25 11:22:13.524161 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:13.524047    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf" event={"ID":"f474090b-08ff-4e37-bb28-093556a57b03","Type":"ContainerDied","Data":"eed98312a4f9083fdebe34d291a53af8afe162df5587fc78d4043e599abf5efa"}
May 25 11:22:14.645780 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.645744    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:22:14.754753 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.754708    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-bundle\") pod \"f474090b-08ff-4e37-bb28-093556a57b03\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") "
May 25 11:22:14.754971 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.754862    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-util\") pod \"f474090b-08ff-4e37-bb28-093556a57b03\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") "
May 25 11:22:14.754971 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.754896    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7bmw\" (UniqueName: \"kubernetes.io/projected/f474090b-08ff-4e37-bb28-093556a57b03-kube-api-access-r7bmw\") pod \"f474090b-08ff-4e37-bb28-093556a57b03\" (UID: \"f474090b-08ff-4e37-bb28-093556a57b03\") "
May 25 11:22:14.755408 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.755382    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-bundle" (OuterVolumeSpecName: "bundle") pod "f474090b-08ff-4e37-bb28-093556a57b03" (UID: "f474090b-08ff-4e37-bb28-093556a57b03"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:22:14.757079 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.757049    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f474090b-08ff-4e37-bb28-093556a57b03-kube-api-access-r7bmw" (OuterVolumeSpecName: "kube-api-access-r7bmw") pod "f474090b-08ff-4e37-bb28-093556a57b03" (UID: "f474090b-08ff-4e37-bb28-093556a57b03"). InnerVolumeSpecName "kube-api-access-r7bmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:22:14.761385 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.761359    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-util" (OuterVolumeSpecName: "util") pod "f474090b-08ff-4e37-bb28-093556a57b03" (UID: "f474090b-08ff-4e37-bb28-093556a57b03"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:22:14.856270 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.856193    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-util\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:22:14.856270 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.856230    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7bmw\" (UniqueName: \"kubernetes.io/projected/f474090b-08ff-4e37-bb28-093556a57b03-kube-api-access-r7bmw\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:22:14.856270 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:14.856242    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f474090b-08ff-4e37-bb28-093556a57b03-bundle\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:22:15.530997 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:15.530960    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf" event={"ID":"f474090b-08ff-4e37-bb28-093556a57b03","Type":"ContainerDied","Data":"e75ae2a82191e9adb0d1c0dfe3ecfd4b95d786bbfb188f59c88454c100c505c5"}
May 25 11:22:15.530997 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:15.530991    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e75ae2a82191e9adb0d1c0dfe3ecfd4b95d786bbfb188f59c88454c100c505c5"
May 25 11:22:15.530997 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:15.530993    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cr5qqf"
May 25 11:22:20.770667 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.770631    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"]
May 25 11:22:20.771056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.770892    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f474090b-08ff-4e37-bb28-093556a57b03" containerName="util"
May 25 11:22:20.771056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.770905    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f474090b-08ff-4e37-bb28-093556a57b03" containerName="util"
May 25 11:22:20.771056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.770920    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f474090b-08ff-4e37-bb28-093556a57b03" containerName="extract"
May 25 11:22:20.771056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.770925    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f474090b-08ff-4e37-bb28-093556a57b03" containerName="extract"
May 25 11:22:20.771056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.770933    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f474090b-08ff-4e37-bb28-093556a57b03" containerName="pull"
May 25 11:22:20.771056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.770938    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f474090b-08ff-4e37-bb28-093556a57b03" containerName="pull"
May 25 11:22:20.771056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.770976    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="f474090b-08ff-4e37-bb28-093556a57b03" containerName="extract"
May 25 11:22:20.775824 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.775808    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:22:20.779157 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.779130    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 11:22:20.779157 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.779150    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 11:22:20.779312 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.779209    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 11:22:20.779312 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.779238    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-h48kk\""
May 25 11:22:20.790309 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.790289    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"]
May 25 11:22:20.898270 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.898238    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c0d34924-ad61-4623-8c82-b00243c7faa7-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c\" (UID: \"c0d34924-ad61-4623-8c82-b00243c7faa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:22:20.898420 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.898290    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxjm\" (UniqueName: \"kubernetes.io/projected/c0d34924-ad61-4623-8c82-b00243c7faa7-kube-api-access-mqxjm\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c\" (UID: \"c0d34924-ad61-4623-8c82-b00243c7faa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:22:20.998882 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.998852    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c0d34924-ad61-4623-8c82-b00243c7faa7-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c\" (UID: \"c0d34924-ad61-4623-8c82-b00243c7faa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:22:20.999065 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:20.998897    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxjm\" (UniqueName: \"kubernetes.io/projected/c0d34924-ad61-4623-8c82-b00243c7faa7-kube-api-access-mqxjm\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c\" (UID: \"c0d34924-ad61-4623-8c82-b00243c7faa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:22:21.001194 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:21.001165    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c0d34924-ad61-4623-8c82-b00243c7faa7-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c\" (UID: \"c0d34924-ad61-4623-8c82-b00243c7faa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:22:21.007399 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:21.007370    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxjm\" (UniqueName: \"kubernetes.io/projected/c0d34924-ad61-4623-8c82-b00243c7faa7-kube-api-access-mqxjm\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c\" (UID: \"c0d34924-ad61-4623-8c82-b00243c7faa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:22:21.085870 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:21.085804    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:22:21.207617 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:21.207585    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"]
May 25 11:22:21.211352 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:22:21.211330    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d34924_ad61_4623_8c82_b00243c7faa7.slice/crio-858569b65400edf77b4b0de72f7be4f70196962ad0af31cfc62c92ebbdff9bf4 WatchSource:0}: Error finding container 858569b65400edf77b4b0de72f7be4f70196962ad0af31cfc62c92ebbdff9bf4: Status 404 returned error can't find the container with id 858569b65400edf77b4b0de72f7be4f70196962ad0af31cfc62c92ebbdff9bf4
May 25 11:22:21.213003 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:21.212984    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:22:21.550058 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:21.550018    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c" event={"ID":"c0d34924-ad61-4623-8c82-b00243c7faa7","Type":"ContainerStarted","Data":"858569b65400edf77b4b0de72f7be4f70196962ad0af31cfc62c92ebbdff9bf4"}
May 25 11:22:27.567402 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:27.567366    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c" event={"ID":"c0d34924-ad61-4623-8c82-b00243c7faa7","Type":"ContainerStarted","Data":"250c53d06e2f4016c6488af744cce92aa9a2f4e300d8541e96edb9491903ef5e"}
May 25 11:22:27.567839 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:27.567479    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:22:27.614696 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:27.614651    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c" podStartSLOduration=2.271919141 podStartE2EDuration="7.614637571s" podCreationTimestamp="2026-05-25 11:22:20 +0000 UTC" firstStartedPulling="2026-05-25 11:22:21.213151561 +0000 UTC m=+309.209326352" lastFinishedPulling="2026-05-25 11:22:26.555869991 +0000 UTC m=+314.552044782" observedRunningTime="2026-05-25 11:22:27.614394333 +0000 UTC m=+315.610569149" watchObservedRunningTime="2026-05-25 11:22:27.614637571 +0000 UTC m=+315.610812384"
May 25 11:22:48.571628 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:22:48.571597    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-z5c6c"
May 25 11:23:33.249851 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.249818    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"]
May 25 11:23:33.252713 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.252698    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:23:33.256165 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.256142    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 11:23:33.256165 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.256157    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 25 11:23:33.256323 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.256185    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-h2xmb\""
May 25 11:23:33.257153 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.257132    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 11:23:33.261439 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.261417    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"]
May 25 11:23:33.371617 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.371585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsw5\" (UniqueName: \"kubernetes.io/projected/99202b1f-c650-49fb-8bac-201064b5c7ef-kube-api-access-knsw5\") pod \"llmisvc-controller-manager-6b94ff949c-c2xgv\" (UID: \"99202b1f-c650-49fb-8bac-201064b5c7ef\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:23:33.371762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.371628    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99202b1f-c650-49fb-8bac-201064b5c7ef-cert\") pod \"llmisvc-controller-manager-6b94ff949c-c2xgv\" (UID: \"99202b1f-c650-49fb-8bac-201064b5c7ef\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:23:33.472627 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.472601    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99202b1f-c650-49fb-8bac-201064b5c7ef-cert\") pod \"llmisvc-controller-manager-6b94ff949c-c2xgv\" (UID: \"99202b1f-c650-49fb-8bac-201064b5c7ef\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:23:33.472733 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.472653    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-knsw5\" (UniqueName: \"kubernetes.io/projected/99202b1f-c650-49fb-8bac-201064b5c7ef-kube-api-access-knsw5\") pod \"llmisvc-controller-manager-6b94ff949c-c2xgv\" (UID: \"99202b1f-c650-49fb-8bac-201064b5c7ef\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:23:33.474939 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.474922    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99202b1f-c650-49fb-8bac-201064b5c7ef-cert\") pod \"llmisvc-controller-manager-6b94ff949c-c2xgv\" (UID: \"99202b1f-c650-49fb-8bac-201064b5c7ef\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:23:33.480343 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.480320    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsw5\" (UniqueName: \"kubernetes.io/projected/99202b1f-c650-49fb-8bac-201064b5c7ef-kube-api-access-knsw5\") pod \"llmisvc-controller-manager-6b94ff949c-c2xgv\" (UID: \"99202b1f-c650-49fb-8bac-201064b5c7ef\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:23:33.563496 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.563439    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:23:33.700591 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.700573    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"]
May 25 11:23:33.703142 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:23:33.703112    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod99202b1f_c650_49fb_8bac_201064b5c7ef.slice/crio-ef641a2b1e53d007aff37cc0e4380b3eb80847b9ca8a10b99f88b532a9d45b7e WatchSource:0}: Error finding container ef641a2b1e53d007aff37cc0e4380b3eb80847b9ca8a10b99f88b532a9d45b7e: Status 404 returned error can't find the container with id ef641a2b1e53d007aff37cc0e4380b3eb80847b9ca8a10b99f88b532a9d45b7e
May 25 11:23:33.734819 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:33.734791    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv" event={"ID":"99202b1f-c650-49fb-8bac-201064b5c7ef","Type":"ContainerStarted","Data":"ef641a2b1e53d007aff37cc0e4380b3eb80847b9ca8a10b99f88b532a9d45b7e"}
May 25 11:23:36.745292 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:36.745252    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv" event={"ID":"99202b1f-c650-49fb-8bac-201064b5c7ef","Type":"ContainerStarted","Data":"5ce0642d0b7adc341bb8dd288e98a20107edfeec858646cceccd078294550c02"}
May 25 11:23:36.745660 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:36.745466    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:23:36.763578 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:23:36.763531    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv" podStartSLOduration=1.600879674 podStartE2EDuration="3.763518952s" podCreationTimestamp="2026-05-25 11:23:33 +0000 UTC" firstStartedPulling="2026-05-25 11:23:33.70424768 +0000 UTC m=+381.700422471" lastFinishedPulling="2026-05-25 11:23:35.866886955 +0000 UTC m=+383.863061749" observedRunningTime="2026-05-25 11:23:36.762969789 +0000 UTC m=+384.759144613" watchObservedRunningTime="2026-05-25 11:23:36.763518952 +0000 UTC m=+384.759693802"
May 25 11:24:07.750497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:07.750464    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-c2xgv"
May 25 11:24:43.100203 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.100128    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-plw64"]
May 25 11:24:43.103191 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.103170    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:43.105739 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.105723    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 25 11:24:43.105939 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.105923    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-2k4xv\""
May 25 11:24:43.114233 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.114213    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-plw64"]
May 25 11:24:43.261883 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.261849    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f859f16-b8d9-4269-8937-a27c56583a24-tls-certs\") pod \"model-serving-api-86f7b4b499-plw64\" (UID: \"2f859f16-b8d9-4269-8937-a27c56583a24\") " pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:43.262046 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.261909    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2h4\" (UniqueName: \"kubernetes.io/projected/2f859f16-b8d9-4269-8937-a27c56583a24-kube-api-access-zf2h4\") pod \"model-serving-api-86f7b4b499-plw64\" (UID: \"2f859f16-b8d9-4269-8937-a27c56583a24\") " pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:43.363166 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.363099    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2h4\" (UniqueName: \"kubernetes.io/projected/2f859f16-b8d9-4269-8937-a27c56583a24-kube-api-access-zf2h4\") pod \"model-serving-api-86f7b4b499-plw64\" (UID: \"2f859f16-b8d9-4269-8937-a27c56583a24\") " pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:43.363166 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.363151    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f859f16-b8d9-4269-8937-a27c56583a24-tls-certs\") pod \"model-serving-api-86f7b4b499-plw64\" (UID: \"2f859f16-b8d9-4269-8937-a27c56583a24\") " pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:43.363319 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:24:43.363239    2569 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
May 25 11:24:43.363319 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:24:43.363313    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f859f16-b8d9-4269-8937-a27c56583a24-tls-certs podName:2f859f16-b8d9-4269-8937-a27c56583a24 nodeName:}" failed. No retries permitted until 2026-05-25 11:24:43.863297876 +0000 UTC m=+451.859472666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2f859f16-b8d9-4269-8937-a27c56583a24-tls-certs") pod "model-serving-api-86f7b4b499-plw64" (UID: "2f859f16-b8d9-4269-8937-a27c56583a24") : secret "model-serving-api-tls" not found
May 25 11:24:43.372470 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.372444    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2h4\" (UniqueName: \"kubernetes.io/projected/2f859f16-b8d9-4269-8937-a27c56583a24-kube-api-access-zf2h4\") pod \"model-serving-api-86f7b4b499-plw64\" (UID: \"2f859f16-b8d9-4269-8937-a27c56583a24\") " pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:43.866443 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.866397    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f859f16-b8d9-4269-8937-a27c56583a24-tls-certs\") pod \"model-serving-api-86f7b4b499-plw64\" (UID: \"2f859f16-b8d9-4269-8937-a27c56583a24\") " pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:43.868693 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:43.868675    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f859f16-b8d9-4269-8937-a27c56583a24-tls-certs\") pod \"model-serving-api-86f7b4b499-plw64\" (UID: \"2f859f16-b8d9-4269-8937-a27c56583a24\") " pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:44.013424 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:44.013380    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:44.133274 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:44.133213    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-plw64"]
May 25 11:24:44.136214 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:24:44.136190    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f859f16_b8d9_4269_8937_a27c56583a24.slice/crio-553db92b4b685f701d4bd3fddfd5d52103dcd819e921b5b195bee7a4cac573f0 WatchSource:0}: Error finding container 553db92b4b685f701d4bd3fddfd5d52103dcd819e921b5b195bee7a4cac573f0: Status 404 returned error can't find the container with id 553db92b4b685f701d4bd3fddfd5d52103dcd819e921b5b195bee7a4cac573f0
May 25 11:24:44.922955 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:44.922916    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-plw64" event={"ID":"2f859f16-b8d9-4269-8937-a27c56583a24","Type":"ContainerStarted","Data":"553db92b4b685f701d4bd3fddfd5d52103dcd819e921b5b195bee7a4cac573f0"}
May 25 11:24:46.929513 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:46.929479    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-plw64" event={"ID":"2f859f16-b8d9-4269-8937-a27c56583a24","Type":"ContainerStarted","Data":"88e7f31bb83bd07716fdf711fb0eecc6f31a0e7102c326db9d2b1defc30e4f94"}
May 25 11:24:46.929894 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:46.929533    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:46.947003 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:46.946957    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-plw64" podStartSLOduration=1.8018896629999999 podStartE2EDuration="3.946944514s" podCreationTimestamp="2026-05-25 11:24:43 +0000 UTC" firstStartedPulling="2026-05-25 11:24:44.137943176 +0000 UTC m=+452.134117966" lastFinishedPulling="2026-05-25 11:24:46.282998023 +0000 UTC m=+454.279172817" observedRunningTime="2026-05-25 11:24:46.946027294 +0000 UTC m=+454.942202108" watchObservedRunningTime="2026-05-25 11:24:46.946944514 +0000 UTC m=+454.943119327"
May 25 11:24:57.937207 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:57.937168    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-plw64"
May 25 11:24:58.641453 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:58.641420    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-nhl42"]
May 25 11:24:58.644456 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:58.644439    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nhl42"
May 25 11:24:58.648471 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:58.648451    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-g9pjq\""
May 25 11:24:58.648700 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:58.648683    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 25 11:24:58.664301 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:58.664272    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nhl42"]
May 25 11:24:58.668296 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:58.668269    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42z55\" (UniqueName: \"kubernetes.io/projected/fcc960dc-a84c-442d-a302-5142a16452db-kube-api-access-42z55\") pod \"s3-init-nhl42\" (UID: \"fcc960dc-a84c-442d-a302-5142a16452db\") " pod="kserve/s3-init-nhl42"
May 25 11:24:58.769345 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:58.769291    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-42z55\" (UniqueName: \"kubernetes.io/projected/fcc960dc-a84c-442d-a302-5142a16452db-kube-api-access-42z55\") pod \"s3-init-nhl42\" (UID: \"fcc960dc-a84c-442d-a302-5142a16452db\") " pod="kserve/s3-init-nhl42"
May 25 11:24:58.790845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:58.790821    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42z55\" (UniqueName: \"kubernetes.io/projected/fcc960dc-a84c-442d-a302-5142a16452db-kube-api-access-42z55\") pod \"s3-init-nhl42\" (UID: \"fcc960dc-a84c-442d-a302-5142a16452db\") " pod="kserve/s3-init-nhl42"
May 25 11:24:58.953015 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:58.952941    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nhl42"
May 25 11:24:59.075670 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:59.075640    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nhl42"]
May 25 11:24:59.078996 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:24:59.078971    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcc960dc_a84c_442d_a302_5142a16452db.slice/crio-4aba05ee7827ed09c9d8380974e2736953f612ddeb8f47fe0c2bbe6013350dd5 WatchSource:0}: Error finding container 4aba05ee7827ed09c9d8380974e2736953f612ddeb8f47fe0c2bbe6013350dd5: Status 404 returned error can't find the container with id 4aba05ee7827ed09c9d8380974e2736953f612ddeb8f47fe0c2bbe6013350dd5
May 25 11:24:59.971461 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:24:59.971413    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nhl42" event={"ID":"fcc960dc-a84c-442d-a302-5142a16452db","Type":"ContainerStarted","Data":"4aba05ee7827ed09c9d8380974e2736953f612ddeb8f47fe0c2bbe6013350dd5"}
May 25 11:25:03.986385 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:03.986345    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nhl42" event={"ID":"fcc960dc-a84c-442d-a302-5142a16452db","Type":"ContainerStarted","Data":"c0861eb43c3f10407c93fbad2de1cddee7f63c499b7aa486941640003a00eea2"}
May 25 11:25:04.005247 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:04.005199    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-nhl42" podStartSLOduration=1.659676272 podStartE2EDuration="6.005185941s" podCreationTimestamp="2026-05-25 11:24:58 +0000 UTC" firstStartedPulling="2026-05-25 11:24:59.080694585 +0000 UTC m=+467.076869376" lastFinishedPulling="2026-05-25 11:25:03.426204251 +0000 UTC m=+471.422379045" observedRunningTime="2026-05-25 11:25:04.004576957 +0000 UTC m=+472.000751771" watchObservedRunningTime="2026-05-25 11:25:04.005185941 +0000 UTC m=+472.001360760"
May 25 11:25:06.995839 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:06.995806    2569 generic.go:358] "Generic (PLEG): container finished" podID="fcc960dc-a84c-442d-a302-5142a16452db" containerID="c0861eb43c3f10407c93fbad2de1cddee7f63c499b7aa486941640003a00eea2" exitCode=0
May 25 11:25:06.996205 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:06.995864    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nhl42" event={"ID":"fcc960dc-a84c-442d-a302-5142a16452db","Type":"ContainerDied","Data":"c0861eb43c3f10407c93fbad2de1cddee7f63c499b7aa486941640003a00eea2"}
May 25 11:25:08.117445 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:08.117420    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nhl42"
May 25 11:25:08.141219 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:08.141197    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42z55\" (UniqueName: \"kubernetes.io/projected/fcc960dc-a84c-442d-a302-5142a16452db-kube-api-access-42z55\") pod \"fcc960dc-a84c-442d-a302-5142a16452db\" (UID: \"fcc960dc-a84c-442d-a302-5142a16452db\") "
May 25 11:25:08.143450 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:08.143418    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc960dc-a84c-442d-a302-5142a16452db-kube-api-access-42z55" (OuterVolumeSpecName: "kube-api-access-42z55") pod "fcc960dc-a84c-442d-a302-5142a16452db" (UID: "fcc960dc-a84c-442d-a302-5142a16452db"). InnerVolumeSpecName "kube-api-access-42z55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:08.242551 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:08.242523    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42z55\" (UniqueName: \"kubernetes.io/projected/fcc960dc-a84c-442d-a302-5142a16452db-kube-api-access-42z55\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:25:09.002719 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:09.002688    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nhl42" event={"ID":"fcc960dc-a84c-442d-a302-5142a16452db","Type":"ContainerDied","Data":"4aba05ee7827ed09c9d8380974e2736953f612ddeb8f47fe0c2bbe6013350dd5"}
May 25 11:25:09.002719 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:09.002719    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aba05ee7827ed09c9d8380974e2736953f612ddeb8f47fe0c2bbe6013350dd5"
May 25 11:25:09.002969 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:09.002736    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nhl42"
May 25 11:25:25.943172 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:25.943138    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cd66696f-f8fld"]
May 25 11:25:29.560781 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.560745    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"]
May 25 11:25:29.561145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.561013    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc960dc-a84c-442d-a302-5142a16452db" containerName="s3-init"
May 25 11:25:29.561145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.561024    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc960dc-a84c-442d-a302-5142a16452db" containerName="s3-init"
May 25 11:25:29.561145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.561070    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="fcc960dc-a84c-442d-a302-5142a16452db" containerName="s3-init"
May 25 11:25:29.563710 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.563695    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.566180 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.566159    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
May 25 11:25:29.566552 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.566204    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
May 25 11:25:29.566552 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.566273    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-g9pjq\""
May 25 11:25:29.573679 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.573659    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"]
May 25 11:25:29.686345 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.686315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz4qm\" (UniqueName: \"kubernetes.io/projected/87be0d53-f438-486f-9178-e7a1b4833432-kube-api-access-kz4qm\") pod \"seaweedfs-tls-serving-7fd5766db9-vk56t\" (UID: \"87be0d53-f438-486f-9178-e7a1b4833432\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.686484 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.686350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/87be0d53-f438-486f-9178-e7a1b4833432-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-vk56t\" (UID: \"87be0d53-f438-486f-9178-e7a1b4833432\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.686484 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.686439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/87be0d53-f438-486f-9178-e7a1b4833432-data\") pod \"seaweedfs-tls-serving-7fd5766db9-vk56t\" (UID: \"87be0d53-f438-486f-9178-e7a1b4833432\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.786948 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.786921    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/87be0d53-f438-486f-9178-e7a1b4833432-data\") pod \"seaweedfs-tls-serving-7fd5766db9-vk56t\" (UID: \"87be0d53-f438-486f-9178-e7a1b4833432\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.787101 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.786960    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kz4qm\" (UniqueName: \"kubernetes.io/projected/87be0d53-f438-486f-9178-e7a1b4833432-kube-api-access-kz4qm\") pod \"seaweedfs-tls-serving-7fd5766db9-vk56t\" (UID: \"87be0d53-f438-486f-9178-e7a1b4833432\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.787101 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.786987    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/87be0d53-f438-486f-9178-e7a1b4833432-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-vk56t\" (UID: \"87be0d53-f438-486f-9178-e7a1b4833432\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.787356 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.787337    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/87be0d53-f438-486f-9178-e7a1b4833432-data\") pod \"seaweedfs-tls-serving-7fd5766db9-vk56t\" (UID: \"87be0d53-f438-486f-9178-e7a1b4833432\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.789248 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.789228    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/87be0d53-f438-486f-9178-e7a1b4833432-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-vk56t\" (UID: \"87be0d53-f438-486f-9178-e7a1b4833432\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.796582 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.796554    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz4qm\" (UniqueName: \"kubernetes.io/projected/87be0d53-f438-486f-9178-e7a1b4833432-kube-api-access-kz4qm\") pod \"seaweedfs-tls-serving-7fd5766db9-vk56t\" (UID: \"87be0d53-f438-486f-9178-e7a1b4833432\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.873046 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.872968    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"
May 25 11:25:29.995734 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:29.995702    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-vk56t"]
May 25 11:25:29.999431 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:25:29.999407    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87be0d53_f438_486f_9178_e7a1b4833432.slice/crio-398e69e3e12370b9526220045e3c01c2d3e33365910883285b4e97f2a34f2bed WatchSource:0}: Error finding container 398e69e3e12370b9526220045e3c01c2d3e33365910883285b4e97f2a34f2bed: Status 404 returned error can't find the container with id 398e69e3e12370b9526220045e3c01c2d3e33365910883285b4e97f2a34f2bed
May 25 11:25:30.058621 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:30.058591    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t" event={"ID":"87be0d53-f438-486f-9178-e7a1b4833432","Type":"ContainerStarted","Data":"398e69e3e12370b9526220045e3c01c2d3e33365910883285b4e97f2a34f2bed"}
May 25 11:25:33.068470 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:33.068428    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t" event={"ID":"87be0d53-f438-486f-9178-e7a1b4833432","Type":"ContainerStarted","Data":"06e19b9f4887f19a0818e40f65051632f14f4db41568b9d2fffd53025ecf2b9e"}
May 25 11:25:33.089153 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:33.089104    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vk56t" podStartSLOduration=1.21017942 podStartE2EDuration="4.089090014s" podCreationTimestamp="2026-05-25 11:25:29 +0000 UTC" firstStartedPulling="2026-05-25 11:25:30.000585181 +0000 UTC m=+497.996759983" lastFinishedPulling="2026-05-25 11:25:32.879495786 +0000 UTC m=+500.875670577" observedRunningTime="2026-05-25 11:25:33.087092666 +0000 UTC m=+501.083267475" watchObservedRunningTime="2026-05-25 11:25:33.089090014 +0000 UTC m=+501.085264804"
May 25 11:25:33.621459 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:33.621433    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-4gb85"]
May 25 11:25:33.627316 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:33.627289    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4gb85"
May 25 11:25:33.631552 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:33.631521    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4gb85"]
May 25 11:25:33.715277 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:33.715237    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99bh5\" (UniqueName: \"kubernetes.io/projected/33fd850e-0a67-445d-87bf-3a3d28918f4a-kube-api-access-99bh5\") pod \"s3-tls-init-serving-4gb85\" (UID: \"33fd850e-0a67-445d-87bf-3a3d28918f4a\") " pod="kserve/s3-tls-init-serving-4gb85"
May 25 11:25:33.815708 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:33.815677    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99bh5\" (UniqueName: \"kubernetes.io/projected/33fd850e-0a67-445d-87bf-3a3d28918f4a-kube-api-access-99bh5\") pod \"s3-tls-init-serving-4gb85\" (UID: \"33fd850e-0a67-445d-87bf-3a3d28918f4a\") " pod="kserve/s3-tls-init-serving-4gb85"
May 25 11:25:33.824172 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:33.824143    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99bh5\" (UniqueName: \"kubernetes.io/projected/33fd850e-0a67-445d-87bf-3a3d28918f4a-kube-api-access-99bh5\") pod \"s3-tls-init-serving-4gb85\" (UID: \"33fd850e-0a67-445d-87bf-3a3d28918f4a\") " pod="kserve/s3-tls-init-serving-4gb85"
May 25 11:25:33.936681 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:33.936596    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4gb85"
May 25 11:25:34.054783 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:34.054740    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4gb85"]
May 25 11:25:34.057689 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:25:34.057660    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33fd850e_0a67_445d_87bf_3a3d28918f4a.slice/crio-a330dd7dbd7fbad887919e492f1cb1ef8faad5fcf778568a1bb54aeec11aea57 WatchSource:0}: Error finding container a330dd7dbd7fbad887919e492f1cb1ef8faad5fcf778568a1bb54aeec11aea57: Status 404 returned error can't find the container with id a330dd7dbd7fbad887919e492f1cb1ef8faad5fcf778568a1bb54aeec11aea57
May 25 11:25:34.071933 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:34.071908    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gb85" event={"ID":"33fd850e-0a67-445d-87bf-3a3d28918f4a","Type":"ContainerStarted","Data":"a330dd7dbd7fbad887919e492f1cb1ef8faad5fcf778568a1bb54aeec11aea57"}
May 25 11:25:35.075694 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:35.075660    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gb85" event={"ID":"33fd850e-0a67-445d-87bf-3a3d28918f4a","Type":"ContainerStarted","Data":"1e265711fd1b6498bdb53c6364b8107aa6c9e7cba0ab53d5385737ab17925c1a"}
May 25 11:25:35.093454 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:35.093407    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-4gb85" podStartSLOduration=2.093393252 podStartE2EDuration="2.093393252s" podCreationTimestamp="2026-05-25 11:25:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:25:35.091970036 +0000 UTC m=+503.088144862" watchObservedRunningTime="2026-05-25 11:25:35.093393252 +0000 UTC m=+503.089568065"
May 25 11:25:38.084466 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:38.084428    2569 generic.go:358] "Generic (PLEG): container finished" podID="33fd850e-0a67-445d-87bf-3a3d28918f4a" containerID="1e265711fd1b6498bdb53c6364b8107aa6c9e7cba0ab53d5385737ab17925c1a" exitCode=0
May 25 11:25:38.084872 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:38.084499    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gb85" event={"ID":"33fd850e-0a67-445d-87bf-3a3d28918f4a","Type":"ContainerDied","Data":"1e265711fd1b6498bdb53c6364b8107aa6c9e7cba0ab53d5385737ab17925c1a"}
May 25 11:25:39.214548 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:39.214525    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4gb85"
May 25 11:25:39.257907 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:39.257881    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99bh5\" (UniqueName: \"kubernetes.io/projected/33fd850e-0a67-445d-87bf-3a3d28918f4a-kube-api-access-99bh5\") pod \"33fd850e-0a67-445d-87bf-3a3d28918f4a\" (UID: \"33fd850e-0a67-445d-87bf-3a3d28918f4a\") "
May 25 11:25:39.259724 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:39.259700    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33fd850e-0a67-445d-87bf-3a3d28918f4a-kube-api-access-99bh5" (OuterVolumeSpecName: "kube-api-access-99bh5") pod "33fd850e-0a67-445d-87bf-3a3d28918f4a" (UID: "33fd850e-0a67-445d-87bf-3a3d28918f4a"). InnerVolumeSpecName "kube-api-access-99bh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:39.358939 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:39.358879    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99bh5\" (UniqueName: \"kubernetes.io/projected/33fd850e-0a67-445d-87bf-3a3d28918f4a-kube-api-access-99bh5\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:25:40.090811 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:40.090763    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gb85" event={"ID":"33fd850e-0a67-445d-87bf-3a3d28918f4a","Type":"ContainerDied","Data":"a330dd7dbd7fbad887919e492f1cb1ef8faad5fcf778568a1bb54aeec11aea57"}
May 25 11:25:40.090811 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:40.090813    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a330dd7dbd7fbad887919e492f1cb1ef8faad5fcf778568a1bb54aeec11aea57"
May 25 11:25:40.090811 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:40.090794    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4gb85"
May 25 11:25:50.961293 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:50.961254    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-58cd66696f-f8fld" podUID="f8bc6690-7d73-4195-9d0a-c25687df97b0" containerName="console" containerID="cri-o://05acf8796719a6c685579c4be99d5b221fe414328582edc5881b75c14f43a9fb" gracePeriod=15
May 25 11:25:51.122690 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.122658    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58cd66696f-f8fld_f8bc6690-7d73-4195-9d0a-c25687df97b0/console/0.log"
May 25 11:25:51.122881 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.122701    2569 generic.go:358] "Generic (PLEG): container finished" podID="f8bc6690-7d73-4195-9d0a-c25687df97b0" containerID="05acf8796719a6c685579c4be99d5b221fe414328582edc5881b75c14f43a9fb" exitCode=2
May 25 11:25:51.122881 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.122740    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cd66696f-f8fld" event={"ID":"f8bc6690-7d73-4195-9d0a-c25687df97b0","Type":"ContainerDied","Data":"05acf8796719a6c685579c4be99d5b221fe414328582edc5881b75c14f43a9fb"}
May 25 11:25:51.199360 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.199339    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58cd66696f-f8fld_f8bc6690-7d73-4195-9d0a-c25687df97b0/console/0.log"
May 25 11:25:51.199463 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.199404    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:25:51.344672 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.344639    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-oauth-config\") pod \"f8bc6690-7d73-4195-9d0a-c25687df97b0\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") "
May 25 11:25:51.344672 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.344674    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-serving-cert\") pod \"f8bc6690-7d73-4195-9d0a-c25687df97b0\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") "
May 25 11:25:51.344930 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.344700    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-oauth-serving-cert\") pod \"f8bc6690-7d73-4195-9d0a-c25687df97b0\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") "
May 25 11:25:51.344930 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.344721    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzv25\" (UniqueName: \"kubernetes.io/projected/f8bc6690-7d73-4195-9d0a-c25687df97b0-kube-api-access-hzv25\") pod \"f8bc6690-7d73-4195-9d0a-c25687df97b0\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") "
May 25 11:25:51.344930 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.344738    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-service-ca\") pod \"f8bc6690-7d73-4195-9d0a-c25687df97b0\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") "
May 25 11:25:51.344930 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.344764    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-trusted-ca-bundle\") pod \"f8bc6690-7d73-4195-9d0a-c25687df97b0\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") "
May 25 11:25:51.344930 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.344817    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-config\") pod \"f8bc6690-7d73-4195-9d0a-c25687df97b0\" (UID: \"f8bc6690-7d73-4195-9d0a-c25687df97b0\") "
May 25 11:25:51.345249 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.345219    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f8bc6690-7d73-4195-9d0a-c25687df97b0" (UID: "f8bc6690-7d73-4195-9d0a-c25687df97b0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:25:51.345307 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.345257    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-config" (OuterVolumeSpecName: "console-config") pod "f8bc6690-7d73-4195-9d0a-c25687df97b0" (UID: "f8bc6690-7d73-4195-9d0a-c25687df97b0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:25:51.345307 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.345268    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f8bc6690-7d73-4195-9d0a-c25687df97b0" (UID: "f8bc6690-7d73-4195-9d0a-c25687df97b0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:25:51.345307 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.345291    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-service-ca" (OuterVolumeSpecName: "service-ca") pod "f8bc6690-7d73-4195-9d0a-c25687df97b0" (UID: "f8bc6690-7d73-4195-9d0a-c25687df97b0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:25:51.346885 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.346847    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f8bc6690-7d73-4195-9d0a-c25687df97b0" (UID: "f8bc6690-7d73-4195-9d0a-c25687df97b0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:25:51.347246 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.347217    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bc6690-7d73-4195-9d0a-c25687df97b0-kube-api-access-hzv25" (OuterVolumeSpecName: "kube-api-access-hzv25") pod "f8bc6690-7d73-4195-9d0a-c25687df97b0" (UID: "f8bc6690-7d73-4195-9d0a-c25687df97b0"). InnerVolumeSpecName "kube-api-access-hzv25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:51.347410 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.347391    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f8bc6690-7d73-4195-9d0a-c25687df97b0" (UID: "f8bc6690-7d73-4195-9d0a-c25687df97b0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:25:51.446041 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.446015    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-oauth-serving-cert\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:25:51.446041 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.446038    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzv25\" (UniqueName: \"kubernetes.io/projected/f8bc6690-7d73-4195-9d0a-c25687df97b0-kube-api-access-hzv25\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:25:51.446041 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.446048    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-service-ca\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:25:51.446230 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.446058    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-trusted-ca-bundle\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:25:51.446230 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.446066    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:25:51.446230 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.446075    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-oauth-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:25:51.446230 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:51.446084    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8bc6690-7d73-4195-9d0a-c25687df97b0-console-serving-cert\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:25:52.127333 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:52.127306    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58cd66696f-f8fld_f8bc6690-7d73-4195-9d0a-c25687df97b0/console/0.log"
May 25 11:25:52.127746 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:52.127357    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cd66696f-f8fld" event={"ID":"f8bc6690-7d73-4195-9d0a-c25687df97b0","Type":"ContainerDied","Data":"eda4bb2209d2ec2bfd619a4c1a0c47443542e76c359fa013e7ab36fc77fb5b4a"}
May 25 11:25:52.127746 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:52.127400    2569 scope.go:117] "RemoveContainer" containerID="05acf8796719a6c685579c4be99d5b221fe414328582edc5881b75c14f43a9fb"
May 25 11:25:52.127746 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:52.127415    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cd66696f-f8fld"
May 25 11:25:52.167720 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:52.167682    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cd66696f-f8fld"]
May 25 11:25:52.169261 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:52.169241    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58cd66696f-f8fld"]
May 25 11:25:52.569778 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:25:52.569748    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8bc6690-7d73-4195-9d0a-c25687df97b0" path="/var/lib/kubelet/pods/f8bc6690-7d73-4195-9d0a-c25687df97b0/volumes"
May 25 11:27:13.032425 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:27:13.032322    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:27:13.036455 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:27:13.033284    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:29:00.340390 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.340357    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"]
May 25 11:29:00.340889 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.340598    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8bc6690-7d73-4195-9d0a-c25687df97b0" containerName="console"
May 25 11:29:00.340889 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.340608    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8bc6690-7d73-4195-9d0a-c25687df97b0" containerName="console"
May 25 11:29:00.340889 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.340627    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33fd850e-0a67-445d-87bf-3a3d28918f4a" containerName="s3-tls-init-serving"
May 25 11:29:00.340889 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.340632    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33fd850e-0a67-445d-87bf-3a3d28918f4a" containerName="s3-tls-init-serving"
May 25 11:29:00.340889 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.340674    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="33fd850e-0a67-445d-87bf-3a3d28918f4a" containerName="s3-tls-init-serving"
May 25 11:29:00.340889 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.340681    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="f8bc6690-7d73-4195-9d0a-c25687df97b0" containerName="console"
May 25 11:29:00.343658 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.343640    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.346285 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.346264    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
May 25 11:29:00.346396 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.346296    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
May 25 11:29:00.346916 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.346898    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:29:00.347369 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.347351    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 11:29:00.347476 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.347384    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:29:00.354466 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.354446    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"]
May 25 11:29:00.443106 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.443083    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ecc625-9b0a-44f6-88fa-21a863972be7-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-4djzb\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.443224 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.443122    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ecc625-9b0a-44f6-88fa-21a863972be7-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-4djzb\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.443224 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.443155    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7t6\" (UniqueName: \"kubernetes.io/projected/14ecc625-9b0a-44f6-88fa-21a863972be7-kube-api-access-2g7t6\") pod \"message-dumper-predictor-c7d86bcbd-4djzb\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.544365 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.544341    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ecc625-9b0a-44f6-88fa-21a863972be7-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-4djzb\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.544449 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.544376    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7t6\" (UniqueName: \"kubernetes.io/projected/14ecc625-9b0a-44f6-88fa-21a863972be7-kube-api-access-2g7t6\") pod \"message-dumper-predictor-c7d86bcbd-4djzb\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.544449 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.544431    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ecc625-9b0a-44f6-88fa-21a863972be7-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-4djzb\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.545108 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.545090    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ecc625-9b0a-44f6-88fa-21a863972be7-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-4djzb\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.546728 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.546712    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ecc625-9b0a-44f6-88fa-21a863972be7-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-4djzb\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.553887 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.553870    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7t6\" (UniqueName: \"kubernetes.io/projected/14ecc625-9b0a-44f6-88fa-21a863972be7-kube-api-access-2g7t6\") pod \"message-dumper-predictor-c7d86bcbd-4djzb\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.653914 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.653842    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:00.774969 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.774939    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"]
May 25 11:29:00.778183 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:29:00.778154    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ecc625_9b0a_44f6_88fa_21a863972be7.slice/crio-e1ffd771155ee07cb716736d7c5f4453b20afff3a9ffe72089254ec2f3e954c8 WatchSource:0}: Error finding container e1ffd771155ee07cb716736d7c5f4453b20afff3a9ffe72089254ec2f3e954c8: Status 404 returned error can't find the container with id e1ffd771155ee07cb716736d7c5f4453b20afff3a9ffe72089254ec2f3e954c8
May 25 11:29:00.779984 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:00.779966    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:29:01.631118 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:01.631078    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb" event={"ID":"14ecc625-9b0a-44f6-88fa-21a863972be7","Type":"ContainerStarted","Data":"e1ffd771155ee07cb716736d7c5f4453b20afff3a9ffe72089254ec2f3e954c8"}
May 25 11:29:02.635727 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:02.635669    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb" event={"ID":"14ecc625-9b0a-44f6-88fa-21a863972be7","Type":"ContainerStarted","Data":"f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2"}
May 25 11:29:04.643235 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:04.643200    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb" event={"ID":"14ecc625-9b0a-44f6-88fa-21a863972be7","Type":"ContainerStarted","Data":"563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87"}
May 25 11:29:04.643597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:04.643437    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:04.643597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:04.643456    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:04.645168 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:04.645147    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:29:04.662926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:04.662882    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb" podStartSLOduration=1.311836146 podStartE2EDuration="4.66286972s" podCreationTimestamp="2026-05-25 11:29:00 +0000 UTC" firstStartedPulling="2026-05-25 11:29:00.780103403 +0000 UTC m=+708.776278208" lastFinishedPulling="2026-05-25 11:29:04.131136979 +0000 UTC m=+712.127311782" observedRunningTime="2026-05-25 11:29:04.661826461 +0000 UTC m=+712.658001277" watchObservedRunningTime="2026-05-25 11:29:04.66286972 +0000 UTC m=+712.659044533"
May 25 11:29:11.652685 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:29:11.652657    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:30:45.434843 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.434732    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-c7d86bcbd-4djzb_14ecc625-9b0a-44f6-88fa-21a863972be7/kserve-container/0.log"
May 25 11:30:45.641185 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.641143    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"]
May 25 11:30:45.641509 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.641468    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb" podUID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerName="kserve-container" containerID="cri-o://f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2" gracePeriod=30
May 25 11:30:45.641597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.641508    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb" podUID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerName="kube-rbac-proxy" containerID="cri-o://563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87" gracePeriod=30
May 25 11:30:45.874389 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.874368    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:30:45.913546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.913514    2569 generic.go:358] "Generic (PLEG): container finished" podID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerID="563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87" exitCode=2
May 25 11:30:45.913546 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.913542    2569 generic.go:358] "Generic (PLEG): container finished" podID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerID="f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2" exitCode=2
May 25 11:30:45.913809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.913592    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"
May 25 11:30:45.913809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.913590    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb" event={"ID":"14ecc625-9b0a-44f6-88fa-21a863972be7","Type":"ContainerDied","Data":"563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87"}
May 25 11:30:45.913809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.913699    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb" event={"ID":"14ecc625-9b0a-44f6-88fa-21a863972be7","Type":"ContainerDied","Data":"f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2"}
May 25 11:30:45.913809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.913709    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb" event={"ID":"14ecc625-9b0a-44f6-88fa-21a863972be7","Type":"ContainerDied","Data":"e1ffd771155ee07cb716736d7c5f4453b20afff3a9ffe72089254ec2f3e954c8"}
May 25 11:30:45.913809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.913723    2569 scope.go:117] "RemoveContainer" containerID="563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87"
May 25 11:30:45.921006 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.920988    2569 scope.go:117] "RemoveContainer" containerID="f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2"
May 25 11:30:45.927737 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.927714    2569 scope.go:117] "RemoveContainer" containerID="563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87"
May 25 11:30:45.928014 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:30:45.927995    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87\": container with ID starting with 563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87 not found: ID does not exist" containerID="563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87"
May 25 11:30:45.928085 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.928020    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87"} err="failed to get container status \"563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87\": rpc error: code = NotFound desc = could not find container \"563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87\": container with ID starting with 563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87 not found: ID does not exist"
May 25 11:30:45.928085 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.928043    2569 scope.go:117] "RemoveContainer" containerID="f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2"
May 25 11:30:45.928285 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:30:45.928269    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2\": container with ID starting with f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2 not found: ID does not exist" containerID="f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2"
May 25 11:30:45.928324 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.928299    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2"} err="failed to get container status \"f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2\": rpc error: code = NotFound desc = could not find container \"f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2\": container with ID starting with f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2 not found: ID does not exist"
May 25 11:30:45.928324 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.928315    2569 scope.go:117] "RemoveContainer" containerID="563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87"
May 25 11:30:45.928530 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.928510    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87"} err="failed to get container status \"563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87\": rpc error: code = NotFound desc = could not find container \"563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87\": container with ID starting with 563cb6a03d7617307ed351d1f881637c5d1378ad6a94f6a437d8e469417f3b87 not found: ID does not exist"
May 25 11:30:45.928586 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.928533    2569 scope.go:117] "RemoveContainer" containerID="f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2"
May 25 11:30:45.928741 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.928721    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2"} err="failed to get container status \"f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2\": rpc error: code = NotFound desc = could not find container \"f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2\": container with ID starting with f18b0b50b7967cfc95d4d5c1c7a21eaa7d4d20bd256e6ba7aef40d4e7a1938c2 not found: ID does not exist"
May 25 11:30:45.968228 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.968195    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ecc625-9b0a-44f6-88fa-21a863972be7-proxy-tls\") pod \"14ecc625-9b0a-44f6-88fa-21a863972be7\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") "
May 25 11:30:45.968366 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.968236    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ecc625-9b0a-44f6-88fa-21a863972be7-message-dumper-kube-rbac-proxy-sar-config\") pod \"14ecc625-9b0a-44f6-88fa-21a863972be7\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") "
May 25 11:30:45.968366 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.968291    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g7t6\" (UniqueName: \"kubernetes.io/projected/14ecc625-9b0a-44f6-88fa-21a863972be7-kube-api-access-2g7t6\") pod \"14ecc625-9b0a-44f6-88fa-21a863972be7\" (UID: \"14ecc625-9b0a-44f6-88fa-21a863972be7\") "
May 25 11:30:45.968600 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.968568    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ecc625-9b0a-44f6-88fa-21a863972be7-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "14ecc625-9b0a-44f6-88fa-21a863972be7" (UID: "14ecc625-9b0a-44f6-88fa-21a863972be7"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:30:45.970422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.970402    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ecc625-9b0a-44f6-88fa-21a863972be7-kube-api-access-2g7t6" (OuterVolumeSpecName: "kube-api-access-2g7t6") pod "14ecc625-9b0a-44f6-88fa-21a863972be7" (UID: "14ecc625-9b0a-44f6-88fa-21a863972be7"). InnerVolumeSpecName "kube-api-access-2g7t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:30:45.970494 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:45.970471    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ecc625-9b0a-44f6-88fa-21a863972be7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14ecc625-9b0a-44f6-88fa-21a863972be7" (UID: "14ecc625-9b0a-44f6-88fa-21a863972be7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:30:46.068935 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:46.068850    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14ecc625-9b0a-44f6-88fa-21a863972be7-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:30:46.068935 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:46.068879    2569 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14ecc625-9b0a-44f6-88fa-21a863972be7-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:30:46.068935 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:46.068890    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g7t6\" (UniqueName: \"kubernetes.io/projected/14ecc625-9b0a-44f6-88fa-21a863972be7-kube-api-access-2g7t6\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:30:46.243266 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:46.243236    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"]
May 25 11:30:46.246329 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:46.246306    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-4djzb"]
May 25 11:30:46.571978 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:30:46.569866    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14ecc625-9b0a-44f6-88fa-21a863972be7" path="/var/lib/kubelet/pods/14ecc625-9b0a-44f6-88fa-21a863972be7/volumes"
May 25 11:32:13.051367 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:32:13.051256    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:32:13.056360 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:32:13.052537    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:36:16.656147 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.656110    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"]
May 25 11:36:16.656742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.656455    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerName="kube-rbac-proxy"
May 25 11:36:16.656742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.656472    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerName="kube-rbac-proxy"
May 25 11:36:16.656742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.656491    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerName="kserve-container"
May 25 11:36:16.656742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.656500    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerName="kserve-container"
May 25 11:36:16.656742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.656560    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerName="kube-rbac-proxy"
May 25 11:36:16.656742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.656576    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="14ecc625-9b0a-44f6-88fa-21a863972be7" containerName="kserve-container"
May 25 11:36:16.659759 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.659734    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.662292 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.662270    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
May 25 11:36:16.662852 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.662835    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
May 25 11:36:16.663488 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.663472    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:36:16.663562 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.663475    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:36:16.663562 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.663533    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 11:36:16.678251 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.678230    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"]
May 25 11:36:16.830613 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.830584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/33654782-df8b-4db7-8577-3f89c481c57a-kube-api-access-dvt6n\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.830786 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.830619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33654782-df8b-4db7-8577-3f89c481c57a-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.830786 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.830657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33654782-df8b-4db7-8577-3f89c481c57a-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.830786 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.830684    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33654782-df8b-4db7-8577-3f89c481c57a-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.931561 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.931495    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33654782-df8b-4db7-8577-3f89c481c57a-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.931561 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.931532    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33654782-df8b-4db7-8577-3f89c481c57a-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.931561 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.931563    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/33654782-df8b-4db7-8577-3f89c481c57a-kube-api-access-dvt6n\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.931791 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.931584    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33654782-df8b-4db7-8577-3f89c481c57a-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.931930 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.931909    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33654782-df8b-4db7-8577-3f89c481c57a-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.932225 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.932203    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33654782-df8b-4db7-8577-3f89c481c57a-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.933889 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.933873    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33654782-df8b-4db7-8577-3f89c481c57a-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.940965 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.940943    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/33654782-df8b-4db7-8577-3f89c481c57a-kube-api-access-dvt6n\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:16.969876 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:16.969854    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:17.096262 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:17.096231    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"]
May 25 11:36:17.099098 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:36:17.099073    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33654782_df8b_4db7_8577_3f89c481c57a.slice/crio-ebce5dee379fb1f775445e6f6018b1eaed64c81c86109a9baff710a5a5a70023 WatchSource:0}: Error finding container ebce5dee379fb1f775445e6f6018b1eaed64c81c86109a9baff710a5a5a70023: Status 404 returned error can't find the container with id ebce5dee379fb1f775445e6f6018b1eaed64c81c86109a9baff710a5a5a70023
May 25 11:36:17.100934 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:17.100914    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:36:17.803353 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:17.803300    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" event={"ID":"33654782-df8b-4db7-8577-3f89c481c57a","Type":"ContainerStarted","Data":"ebce5dee379fb1f775445e6f6018b1eaed64c81c86109a9baff710a5a5a70023"}
May 25 11:36:21.816479 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:21.816441    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" event={"ID":"33654782-df8b-4db7-8577-3f89c481c57a","Type":"ContainerStarted","Data":"a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579"}
May 25 11:36:24.824509 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:24.824480    2569 generic.go:358] "Generic (PLEG): container finished" podID="33654782-df8b-4db7-8577-3f89c481c57a" containerID="a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579" exitCode=0
May 25 11:36:24.824885 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:24.824550    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" event={"ID":"33654782-df8b-4db7-8577-3f89c481c57a","Type":"ContainerDied","Data":"a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579"}
May 25 11:36:31.849528 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:31.849490    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" event={"ID":"33654782-df8b-4db7-8577-3f89c481c57a","Type":"ContainerStarted","Data":"21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0"}
May 25 11:36:31.849528 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:31.849527    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" event={"ID":"33654782-df8b-4db7-8577-3f89c481c57a","Type":"ContainerStarted","Data":"d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb"}
May 25 11:36:31.850113 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:31.849817    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:31.872991 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:31.872946    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" podStartSLOduration=2.072446084 podStartE2EDuration="15.872931457s" podCreationTimestamp="2026-05-25 11:36:16 +0000 UTC" firstStartedPulling="2026-05-25 11:36:17.101115802 +0000 UTC m=+1145.097290607" lastFinishedPulling="2026-05-25 11:36:30.901601189 +0000 UTC m=+1158.897775980" observedRunningTime="2026-05-25 11:36:31.872503516 +0000 UTC m=+1159.868678330" watchObservedRunningTime="2026-05-25 11:36:31.872931457 +0000 UTC m=+1159.869106270"
May 25 11:36:32.852329 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:32.852298    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:32.853588 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:32.853549    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
May 25 11:36:33.855309 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:33.855266    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
May 25 11:36:38.859642 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:38.859613    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:38.860745 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:38.860723    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:46.682909 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:46.682869    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"]
May 25 11:36:46.683390 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:46.683278    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kserve-container" containerID="cri-o://d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb" gracePeriod=30
May 25 11:36:46.683390 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:46.683339    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kube-rbac-proxy" containerID="cri-o://21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0" gracePeriod=30
May 25 11:36:46.890964 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:46.890927    2569 generic.go:358] "Generic (PLEG): container finished" podID="33654782-df8b-4db7-8577-3f89c481c57a" containerID="21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0" exitCode=2
May 25 11:36:46.891126 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:46.891009    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" event={"ID":"33654782-df8b-4db7-8577-3f89c481c57a","Type":"ContainerDied","Data":"21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0"}
May 25 11:36:47.335567 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.335548    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:47.345381 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.345362    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33654782-df8b-4db7-8577-3f89c481c57a-kserve-provision-location\") pod \"33654782-df8b-4db7-8577-3f89c481c57a\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") "
May 25 11:36:47.345497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.345407    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33654782-df8b-4db7-8577-3f89c481c57a-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"33654782-df8b-4db7-8577-3f89c481c57a\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") "
May 25 11:36:47.345497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.345456    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/33654782-df8b-4db7-8577-3f89c481c57a-kube-api-access-dvt6n\") pod \"33654782-df8b-4db7-8577-3f89c481c57a\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") "
May 25 11:36:47.345497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.345477    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33654782-df8b-4db7-8577-3f89c481c57a-proxy-tls\") pod \"33654782-df8b-4db7-8577-3f89c481c57a\" (UID: \"33654782-df8b-4db7-8577-3f89c481c57a\") "
May 25 11:36:47.345742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.345719    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33654782-df8b-4db7-8577-3f89c481c57a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33654782-df8b-4db7-8577-3f89c481c57a" (UID: "33654782-df8b-4db7-8577-3f89c481c57a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:36:47.345812 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.345785    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33654782-df8b-4db7-8577-3f89c481c57a-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "33654782-df8b-4db7-8577-3f89c481c57a" (UID: "33654782-df8b-4db7-8577-3f89c481c57a"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:36:47.347471 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.347444    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33654782-df8b-4db7-8577-3f89c481c57a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33654782-df8b-4db7-8577-3f89c481c57a" (UID: "33654782-df8b-4db7-8577-3f89c481c57a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:36:47.347471 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.347449    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33654782-df8b-4db7-8577-3f89c481c57a-kube-api-access-dvt6n" (OuterVolumeSpecName: "kube-api-access-dvt6n") pod "33654782-df8b-4db7-8577-3f89c481c57a" (UID: "33654782-df8b-4db7-8577-3f89c481c57a"). InnerVolumeSpecName "kube-api-access-dvt6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:36:47.446292 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.446221    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33654782-df8b-4db7-8577-3f89c481c57a-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:36:47.446292 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.446247    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33654782-df8b-4db7-8577-3f89c481c57a-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:36:47.446292 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.446258    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/33654782-df8b-4db7-8577-3f89c481c57a-kube-api-access-dvt6n\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:36:47.446292 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.446267    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33654782-df8b-4db7-8577-3f89c481c57a-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:36:47.895790 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.895739    2569 generic.go:358] "Generic (PLEG): container finished" podID="33654782-df8b-4db7-8577-3f89c481c57a" containerID="d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb" exitCode=0
May 25 11:36:47.896238 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.895815    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" event={"ID":"33654782-df8b-4db7-8577-3f89c481c57a","Type":"ContainerDied","Data":"d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb"}
May 25 11:36:47.896238 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.895854    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj" event={"ID":"33654782-df8b-4db7-8577-3f89c481c57a","Type":"ContainerDied","Data":"ebce5dee379fb1f775445e6f6018b1eaed64c81c86109a9baff710a5a5a70023"}
May 25 11:36:47.896238 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.895870    2569 scope.go:117] "RemoveContainer" containerID="21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0"
May 25 11:36:47.896238 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.895873    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"
May 25 11:36:47.904126 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.903780    2569 scope.go:117] "RemoveContainer" containerID="d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb"
May 25 11:36:47.910939 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.910920    2569 scope.go:117] "RemoveContainer" containerID="a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579"
May 25 11:36:47.917174 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.917156    2569 scope.go:117] "RemoveContainer" containerID="21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0"
May 25 11:36:47.917463 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:36:47.917444    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0\": container with ID starting with 21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0 not found: ID does not exist" containerID="21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0"
May 25 11:36:47.917552 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.917480    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0"} err="failed to get container status \"21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0\": rpc error: code = NotFound desc = could not find container \"21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0\": container with ID starting with 21ea3918e68d299460038187ebf929b86748cfb5e725d1526cdde7e784db88b0 not found: ID does not exist"
May 25 11:36:47.917552 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.917497    2569 scope.go:117] "RemoveContainer" containerID="d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb"
May 25 11:36:47.917843 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:36:47.917820    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb\": container with ID starting with d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb not found: ID does not exist" containerID="d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb"
May 25 11:36:47.917976 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.917952    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb"} err="failed to get container status \"d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb\": rpc error: code = NotFound desc = could not find container \"d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb\": container with ID starting with d24b9de4755666da46c426fda22d4a8e2850f43f602cdff0cfa6075eba0b49bb not found: ID does not exist"
May 25 11:36:47.918056 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.917978    2569 scope.go:117] "RemoveContainer" containerID="a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579"
May 25 11:36:47.918252 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:36:47.918230    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579\": container with ID starting with a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579 not found: ID does not exist" containerID="a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579"
May 25 11:36:47.918295 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.918260    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579"} err="failed to get container status \"a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579\": rpc error: code = NotFound desc = could not find container \"a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579\": container with ID starting with a15e2ba3bbbae0194fe4620271d247a6d25c725d0f90e791c9b105f471d94579 not found: ID does not exist"
May 25 11:36:47.919230 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.919211    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"]
May 25 11:36:47.924961 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:47.924938    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-f8rwj"]
May 25 11:36:48.569435 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:36:48.569404    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33654782-df8b-4db7-8577-3f89c481c57a" path="/var/lib/kubelet/pods/33654782-df8b-4db7-8577-3f89c481c57a/volumes"
May 25 11:37:13.070001 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:37:13.069973    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:37:13.074705 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:37:13.074682    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:38:57.143073 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.143037    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"]
May 25 11:38:57.143592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.143293    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kube-rbac-proxy"
May 25 11:38:57.143592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.143303    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kube-rbac-proxy"
May 25 11:38:57.143592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.143311    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="storage-initializer"
May 25 11:38:57.143592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.143317    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="storage-initializer"
May 25 11:38:57.143592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.143327    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kserve-container"
May 25 11:38:57.143592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.143334    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kserve-container"
May 25 11:38:57.143592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.143373    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kserve-container"
May 25 11:38:57.143592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.143380    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="33654782-df8b-4db7-8577-3f89c481c57a" containerName="kube-rbac-proxy"
May 25 11:38:57.151129 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.151103    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.153725 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.153705    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
May 25 11:38:57.153854 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.153787    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:38:57.154205 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.154190    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 11:38:57.154950 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.154912    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:38:57.155414 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.155064    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
May 25 11:38:57.157367 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.157345    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"]
May 25 11:38:57.221678 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.221649    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23ab0b86-2fe7-4065-9869-68739249fa1c-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.221857 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.221690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23ab0b86-2fe7-4065-9869-68739249fa1c-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.221857 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.221707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23ab0b86-2fe7-4065-9869-68739249fa1c-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.221857 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.221754    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nclhn\" (UniqueName: \"kubernetes.io/projected/23ab0b86-2fe7-4065-9869-68739249fa1c-kube-api-access-nclhn\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.322350 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.322325    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23ab0b86-2fe7-4065-9869-68739249fa1c-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.322492 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.322357    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23ab0b86-2fe7-4065-9869-68739249fa1c-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.322492 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.322380    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nclhn\" (UniqueName: \"kubernetes.io/projected/23ab0b86-2fe7-4065-9869-68739249fa1c-kube-api-access-nclhn\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.322492 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.322428    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23ab0b86-2fe7-4065-9869-68739249fa1c-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.322818 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.322793    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23ab0b86-2fe7-4065-9869-68739249fa1c-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.323068 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.323049    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23ab0b86-2fe7-4065-9869-68739249fa1c-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.324726 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.324709    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23ab0b86-2fe7-4065-9869-68739249fa1c-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.331146 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.331123    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nclhn\" (UniqueName: \"kubernetes.io/projected/23ab0b86-2fe7-4065-9869-68739249fa1c-kube-api-access-nclhn\") pod \"isvc-paddle-predictor-6b8b7cfb4b-762sq\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.462195 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.462116    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:38:57.582459 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:57.582323    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"]
May 25 11:38:57.585003 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:38:57.584974    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ab0b86_2fe7_4065_9869_68739249fa1c.slice/crio-05b9cfc8c8e41c1895e84253c06345e091da313ba50f51e2e551a725cf8b138e WatchSource:0}: Error finding container 05b9cfc8c8e41c1895e84253c06345e091da313ba50f51e2e551a725cf8b138e: Status 404 returned error can't find the container with id 05b9cfc8c8e41c1895e84253c06345e091da313ba50f51e2e551a725cf8b138e
May 25 11:38:58.267896 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:58.267861    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" event={"ID":"23ab0b86-2fe7-4065-9869-68739249fa1c","Type":"ContainerStarted","Data":"b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717"}
May 25 11:38:58.267896 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:38:58.267895    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" event={"ID":"23ab0b86-2fe7-4065-9869-68739249fa1c","Type":"ContainerStarted","Data":"05b9cfc8c8e41c1895e84253c06345e091da313ba50f51e2e551a725cf8b138e"}
May 25 11:39:02.280597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:02.280509    2569 generic.go:358] "Generic (PLEG): container finished" podID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerID="b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717" exitCode=0
May 25 11:39:02.280597 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:02.280582    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" event={"ID":"23ab0b86-2fe7-4065-9869-68739249fa1c","Type":"ContainerDied","Data":"b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717"}
May 25 11:39:13.324000 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:13.323963    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" event={"ID":"23ab0b86-2fe7-4065-9869-68739249fa1c","Type":"ContainerStarted","Data":"bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229"}
May 25 11:39:13.324377 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:13.324009    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" event={"ID":"23ab0b86-2fe7-4065-9869-68739249fa1c","Type":"ContainerStarted","Data":"36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02"}
May 25 11:39:13.324377 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:13.324229    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:39:14.326588 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:14.326552    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:39:14.327534 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:14.327508    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
May 25 11:39:15.328939 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:15.328905    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
May 25 11:39:20.334723 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:20.334692    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:39:20.335184 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:20.335159    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
May 25 11:39:20.354545 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:20.354489    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podStartSLOduration=12.519871651999999 podStartE2EDuration="23.354473016s" podCreationTimestamp="2026-05-25 11:38:57 +0000 UTC" firstStartedPulling="2026-05-25 11:39:02.281672138 +0000 UTC m=+1310.277846929" lastFinishedPulling="2026-05-25 11:39:13.116273498 +0000 UTC m=+1321.112448293" observedRunningTime="2026-05-25 11:39:13.344313236 +0000 UTC m=+1321.340488049" watchObservedRunningTime="2026-05-25 11:39:20.354473016 +0000 UTC m=+1328.350647829"
May 25 11:39:30.335364 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:30.335328    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
May 25 11:39:40.335636 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:40.335597    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
May 25 11:39:50.335139 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:39:50.335096    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
May 25 11:40:00.336209 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:00.336178    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:40:08.617498 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.617465    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"]
May 25 11:40:08.617929 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.617747    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container" containerID="cri-o://36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02" gracePeriod=30
May 25 11:40:08.617929 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.617814    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kube-rbac-proxy" containerID="cri-o://bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229" gracePeriod=30
May 25 11:40:08.766563 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.766536    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"]
May 25 11:40:08.769646 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.769630    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:08.772755 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.772734    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
May 25 11:40:08.772945 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.772933    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
May 25 11:40:08.784160 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.784138    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"]
May 25 11:40:08.943929 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.943837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19502667-3b22-48ca-bb2c-bae2b3e3c391-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:08.943929 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.943902    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19502667-3b22-48ca-bb2c-bae2b3e3c391-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:08.944132 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.943931    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbxm\" (UniqueName: \"kubernetes.io/projected/19502667-3b22-48ca-bb2c-bae2b3e3c391-kube-api-access-tdbxm\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:08.944132 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:08.943995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19502667-3b22-48ca-bb2c-bae2b3e3c391-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.044733 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.044704    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19502667-3b22-48ca-bb2c-bae2b3e3c391-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.044907 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.044743    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19502667-3b22-48ca-bb2c-bae2b3e3c391-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.044907 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.044874    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19502667-3b22-48ca-bb2c-bae2b3e3c391-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.045025 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.044924    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbxm\" (UniqueName: \"kubernetes.io/projected/19502667-3b22-48ca-bb2c-bae2b3e3c391-kube-api-access-tdbxm\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.045127 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.045106    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19502667-3b22-48ca-bb2c-bae2b3e3c391-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.045484 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.045463    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19502667-3b22-48ca-bb2c-bae2b3e3c391-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.047173 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.047155    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19502667-3b22-48ca-bb2c-bae2b3e3c391-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.055454 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.055435    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbxm\" (UniqueName: \"kubernetes.io/projected/19502667-3b22-48ca-bb2c-bae2b3e3c391-kube-api-access-tdbxm\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.079390 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.079373    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:09.200986 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.200964    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"]
May 25 11:40:09.203187 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:40:09.203161    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19502667_3b22_48ca_bb2c_bae2b3e3c391.slice/crio-c6b1e329e23a5f4899481c0eb681a35a0b5df8873ea7e09325029dc02497ca2b WatchSource:0}: Error finding container c6b1e329e23a5f4899481c0eb681a35a0b5df8873ea7e09325029dc02497ca2b: Status 404 returned error can't find the container with id c6b1e329e23a5f4899481c0eb681a35a0b5df8873ea7e09325029dc02497ca2b
May 25 11:40:09.476458 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.476375    2569 generic.go:358] "Generic (PLEG): container finished" podID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerID="bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229" exitCode=2
May 25 11:40:09.476604 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.476458    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" event={"ID":"23ab0b86-2fe7-4065-9869-68739249fa1c","Type":"ContainerDied","Data":"bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229"}
May 25 11:40:09.477715 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.477693    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" event={"ID":"19502667-3b22-48ca-bb2c-bae2b3e3c391","Type":"ContainerStarted","Data":"7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8"}
May 25 11:40:09.477853 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:09.477718    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" event={"ID":"19502667-3b22-48ca-bb2c-bae2b3e3c391","Type":"ContainerStarted","Data":"c6b1e329e23a5f4899481c0eb681a35a0b5df8873ea7e09325029dc02497ca2b"}
May 25 11:40:10.329812 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:10.329751    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
May 25 11:40:10.335142 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:10.335112    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
May 25 11:40:11.160765 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.160744    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:40:11.262463 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.262430    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23ab0b86-2fe7-4065-9869-68739249fa1c-proxy-tls\") pod \"23ab0b86-2fe7-4065-9869-68739249fa1c\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") "
May 25 11:40:11.262626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.262511    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nclhn\" (UniqueName: \"kubernetes.io/projected/23ab0b86-2fe7-4065-9869-68739249fa1c-kube-api-access-nclhn\") pod \"23ab0b86-2fe7-4065-9869-68739249fa1c\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") "
May 25 11:40:11.262626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.262559    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23ab0b86-2fe7-4065-9869-68739249fa1c-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"23ab0b86-2fe7-4065-9869-68739249fa1c\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") "
May 25 11:40:11.262711 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.262681    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23ab0b86-2fe7-4065-9869-68739249fa1c-kserve-provision-location\") pod \"23ab0b86-2fe7-4065-9869-68739249fa1c\" (UID: \"23ab0b86-2fe7-4065-9869-68739249fa1c\") "
May 25 11:40:11.262957 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.262928    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ab0b86-2fe7-4065-9869-68739249fa1c-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "23ab0b86-2fe7-4065-9869-68739249fa1c" (UID: "23ab0b86-2fe7-4065-9869-68739249fa1c"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:40:11.264603 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.264576    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ab0b86-2fe7-4065-9869-68739249fa1c-kube-api-access-nclhn" (OuterVolumeSpecName: "kube-api-access-nclhn") pod "23ab0b86-2fe7-4065-9869-68739249fa1c" (UID: "23ab0b86-2fe7-4065-9869-68739249fa1c"). InnerVolumeSpecName "kube-api-access-nclhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:40:11.264695 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.264603    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ab0b86-2fe7-4065-9869-68739249fa1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23ab0b86-2fe7-4065-9869-68739249fa1c" (UID: "23ab0b86-2fe7-4065-9869-68739249fa1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:40:11.274412 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.274383    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ab0b86-2fe7-4065-9869-68739249fa1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23ab0b86-2fe7-4065-9869-68739249fa1c" (UID: "23ab0b86-2fe7-4065-9869-68739249fa1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:11.363340 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.363307    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23ab0b86-2fe7-4065-9869-68739249fa1c-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:40:11.363340 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.363334    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23ab0b86-2fe7-4065-9869-68739249fa1c-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:40:11.363340 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.363345    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nclhn\" (UniqueName: \"kubernetes.io/projected/23ab0b86-2fe7-4065-9869-68739249fa1c-kube-api-access-nclhn\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:40:11.363733 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.363357    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23ab0b86-2fe7-4065-9869-68739249fa1c-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:40:11.486679 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.486645    2569 generic.go:358] "Generic (PLEG): container finished" podID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerID="36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02" exitCode=0
May 25 11:40:11.486827 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.486692    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" event={"ID":"23ab0b86-2fe7-4065-9869-68739249fa1c","Type":"ContainerDied","Data":"36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02"}
May 25 11:40:11.486827 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.486714    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq" event={"ID":"23ab0b86-2fe7-4065-9869-68739249fa1c","Type":"ContainerDied","Data":"05b9cfc8c8e41c1895e84253c06345e091da313ba50f51e2e551a725cf8b138e"}
May 25 11:40:11.486827 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.486723    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"
May 25 11:40:11.486977 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.486730    2569 scope.go:117] "RemoveContainer" containerID="bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229"
May 25 11:40:11.494329 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.494310    2569 scope.go:117] "RemoveContainer" containerID="36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02"
May 25 11:40:11.500946 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.500929    2569 scope.go:117] "RemoveContainer" containerID="b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717"
May 25 11:40:11.507441 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.507420    2569 scope.go:117] "RemoveContainer" containerID="bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229"
May 25 11:40:11.507751 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:40:11.507722    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229\": container with ID starting with bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229 not found: ID does not exist" containerID="bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229"
May 25 11:40:11.507845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.507762    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229"} err="failed to get container status \"bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229\": rpc error: code = NotFound desc = could not find container \"bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229\": container with ID starting with bab480ceb6dc802b7dfbc15557e551659635ec22fc8832431d3dc29cbcf75229 not found: ID does not exist"
May 25 11:40:11.507845 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.507794    2569 scope.go:117] "RemoveContainer" containerID="36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02"
May 25 11:40:11.508090 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:40:11.508055    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02\": container with ID starting with 36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02 not found: ID does not exist" containerID="36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02"
May 25 11:40:11.508257 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.508098    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02"} err="failed to get container status \"36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02\": rpc error: code = NotFound desc = could not find container \"36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02\": container with ID starting with 36a557f3425317981521cdbd6978f869289f1cfbbd2894113c8f53a823424f02 not found: ID does not exist"
May 25 11:40:11.508257 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.508121    2569 scope.go:117] "RemoveContainer" containerID="b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717"
May 25 11:40:11.508437 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:40:11.508421    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717\": container with ID starting with b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717 not found: ID does not exist" containerID="b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717"
May 25 11:40:11.508481 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.508442    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717"} err="failed to get container status \"b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717\": rpc error: code = NotFound desc = could not find container \"b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717\": container with ID starting with b2f0ce8f4a9b17ddf6eef4098e78160b3295aff1b28f00b86bd713e24a086717 not found: ID does not exist"
May 25 11:40:11.510034 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.510015    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"]
May 25 11:40:11.516286 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:11.516241    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-762sq"]
May 25 11:40:12.570245 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:12.570212    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" path="/var/lib/kubelet/pods/23ab0b86-2fe7-4065-9869-68739249fa1c/volumes"
May 25 11:40:14.497812 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:14.497753    2569 generic.go:358] "Generic (PLEG): container finished" podID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerID="7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8" exitCode=0
May 25 11:40:14.498202 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:14.497828    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" event={"ID":"19502667-3b22-48ca-bb2c-bae2b3e3c391","Type":"ContainerDied","Data":"7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8"}
May 25 11:40:15.502533 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:15.502500    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" event={"ID":"19502667-3b22-48ca-bb2c-bae2b3e3c391","Type":"ContainerStarted","Data":"12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c"}
May 25 11:40:15.502533 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:15.502537    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" event={"ID":"19502667-3b22-48ca-bb2c-bae2b3e3c391","Type":"ContainerStarted","Data":"0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf"}
May 25 11:40:15.502970 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:15.502729    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:16.505529 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:16.505501    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:16.506596 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:16.506571    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
May 25 11:40:17.508027 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:17.507990    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
May 25 11:40:22.512710 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:22.512680    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:40:22.513292 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:22.513273    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
May 25 11:40:22.532125 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:22.532087    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podStartSLOduration=14.532076544 podStartE2EDuration="14.532076544s" podCreationTimestamp="2026-05-25 11:40:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:40:15.54197779 +0000 UTC m=+1383.538152603" watchObservedRunningTime="2026-05-25 11:40:22.532076544 +0000 UTC m=+1390.528251356"
May 25 11:40:32.513257 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:32.513216    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
May 25 11:40:42.513637 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:42.513599    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
May 25 11:40:52.513523 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:40:52.513484    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
May 25 11:41:02.513989 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:02.513902    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:41:10.083653 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.083619    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"]
May 25 11:41:10.084219 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.083957    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container" containerID="cri-o://0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf" gracePeriod=30
May 25 11:41:10.084219 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.084010    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kube-rbac-proxy" containerID="cri-o://12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c" gracePeriod=30
May 25 11:41:10.184176 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.184143    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"]
May 25 11:41:10.184419 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.184407    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container"
May 25 11:41:10.184470 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.184420    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container"
May 25 11:41:10.184470 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.184432    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kube-rbac-proxy"
May 25 11:41:10.184470 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.184438    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kube-rbac-proxy"
May 25 11:41:10.184470 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.184449    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="storage-initializer"
May 25 11:41:10.184470 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.184455    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="storage-initializer"
May 25 11:41:10.184621 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.184492    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kube-rbac-proxy"
May 25 11:41:10.184621 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.184502    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="23ab0b86-2fe7-4065-9869-68739249fa1c" containerName="kserve-container"
May 25 11:41:10.187412 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.187396    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.189988 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.189966    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
May 25 11:41:10.190099 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.189966    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
May 25 11:41:10.199273 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.199252    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"]
May 25 11:41:10.290029 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.290005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.290189 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.290038    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f49bc95-7fe7-413f-b520-9c1ed31527d1-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.290189 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.290065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5kf\" (UniqueName: \"kubernetes.io/projected/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kube-api-access-xm5kf\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.290189 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.290090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f49bc95-7fe7-413f-b520-9c1ed31527d1-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.390908 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.390826    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f49bc95-7fe7-413f-b520-9c1ed31527d1-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.390908 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.390893    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.391126 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.390915    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f49bc95-7fe7-413f-b520-9c1ed31527d1-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.391126 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.390941    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5kf\" (UniqueName: \"kubernetes.io/projected/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kube-api-access-xm5kf\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.391367 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.391347    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.391976 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.391559    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f49bc95-7fe7-413f-b520-9c1ed31527d1-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.397454 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.397433    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f49bc95-7fe7-413f-b520-9c1ed31527d1-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.400929 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.400910    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5kf\" (UniqueName: \"kubernetes.io/projected/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kube-api-access-xm5kf\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.498012 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.497986    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:10.625061 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.625009    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"]
May 25 11:41:10.627104 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:41:10.627078    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f49bc95_7fe7_413f_b520_9c1ed31527d1.slice/crio-2a2a41e759a11a2cada7f357650ac72b90058466e5cd7039aa61b52f466ebd4b WatchSource:0}: Error finding container 2a2a41e759a11a2cada7f357650ac72b90058466e5cd7039aa61b52f466ebd4b: Status 404 returned error can't find the container with id 2a2a41e759a11a2cada7f357650ac72b90058466e5cd7039aa61b52f466ebd4b
May 25 11:41:10.655696 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.655670    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" event={"ID":"8f49bc95-7fe7-413f-b520-9c1ed31527d1","Type":"ContainerStarted","Data":"2a2a41e759a11a2cada7f357650ac72b90058466e5cd7039aa61b52f466ebd4b"}
May 25 11:41:10.657405 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.657385    2569 generic.go:358] "Generic (PLEG): container finished" podID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerID="12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c" exitCode=2
May 25 11:41:10.657499 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:10.657416    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" event={"ID":"19502667-3b22-48ca-bb2c-bae2b3e3c391","Type":"ContainerDied","Data":"12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c"}
May 25 11:41:11.662205 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:11.662167    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" event={"ID":"8f49bc95-7fe7-413f-b520-9c1ed31527d1","Type":"ContainerStarted","Data":"8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4"}
May 25 11:41:12.508755 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.508719    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
May 25 11:41:12.513310 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.513276    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
May 25 11:41:12.621801 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.621756    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:41:12.667830 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.667797    2569 generic.go:358] "Generic (PLEG): container finished" podID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerID="0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf" exitCode=0
May 25 11:41:12.668159 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.667876    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"
May 25 11:41:12.668159 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.667910    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" event={"ID":"19502667-3b22-48ca-bb2c-bae2b3e3c391","Type":"ContainerDied","Data":"0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf"}
May 25 11:41:12.668159 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.667941    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp" event={"ID":"19502667-3b22-48ca-bb2c-bae2b3e3c391","Type":"ContainerDied","Data":"c6b1e329e23a5f4899481c0eb681a35a0b5df8873ea7e09325029dc02497ca2b"}
May 25 11:41:12.668159 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.667958    2569 scope.go:117] "RemoveContainer" containerID="12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c"
May 25 11:41:12.675123 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.675108    2569 scope.go:117] "RemoveContainer" containerID="0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf"
May 25 11:41:12.681636 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.681621    2569 scope.go:117] "RemoveContainer" containerID="7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8"
May 25 11:41:12.688046 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.688031    2569 scope.go:117] "RemoveContainer" containerID="12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c"
May 25 11:41:12.688283 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:41:12.688266    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c\": container with ID starting with 12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c not found: ID does not exist" containerID="12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c"
May 25 11:41:12.688327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.688290    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c"} err="failed to get container status \"12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c\": rpc error: code = NotFound desc = could not find container \"12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c\": container with ID starting with 12780dcd67bf73c9682343c68e70419eca346c9526926a840607f141932e566c not found: ID does not exist"
May 25 11:41:12.688327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.688307    2569 scope.go:117] "RemoveContainer" containerID="0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf"
May 25 11:41:12.688500 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:41:12.688486    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf\": container with ID starting with 0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf not found: ID does not exist" containerID="0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf"
May 25 11:41:12.688548 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.688503    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf"} err="failed to get container status \"0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf\": rpc error: code = NotFound desc = could not find container \"0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf\": container with ID starting with 0313657738a1dd968820f1f3cfb78bb7e1a29a8c71bd9d112f3df943fbfa6fbf not found: ID does not exist"
May 25 11:41:12.688548 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.688514    2569 scope.go:117] "RemoveContainer" containerID="7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8"
May 25 11:41:12.688700 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:41:12.688686    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8\": container with ID starting with 7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8 not found: ID does not exist" containerID="7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8"
May 25 11:41:12.688739 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.688702    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8"} err="failed to get container status \"7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8\": rpc error: code = NotFound desc = could not find container \"7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8\": container with ID starting with 7df0a710173e25c6a89984bbfb57f46affa0e945d02cc49a2d87d98ba9460ad8 not found: ID does not exist"
May 25 11:41:12.710012 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.709968    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19502667-3b22-48ca-bb2c-bae2b3e3c391-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"19502667-3b22-48ca-bb2c-bae2b3e3c391\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") "
May 25 11:41:12.710012 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.710009    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19502667-3b22-48ca-bb2c-bae2b3e3c391-proxy-tls\") pod \"19502667-3b22-48ca-bb2c-bae2b3e3c391\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") "
May 25 11:41:12.710135 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.710044    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdbxm\" (UniqueName: \"kubernetes.io/projected/19502667-3b22-48ca-bb2c-bae2b3e3c391-kube-api-access-tdbxm\") pod \"19502667-3b22-48ca-bb2c-bae2b3e3c391\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") "
May 25 11:41:12.710135 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.710064    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19502667-3b22-48ca-bb2c-bae2b3e3c391-kserve-provision-location\") pod \"19502667-3b22-48ca-bb2c-bae2b3e3c391\" (UID: \"19502667-3b22-48ca-bb2c-bae2b3e3c391\") "
May 25 11:41:12.710408 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.710374    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19502667-3b22-48ca-bb2c-bae2b3e3c391-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "19502667-3b22-48ca-bb2c-bae2b3e3c391" (UID: "19502667-3b22-48ca-bb2c-bae2b3e3c391"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:41:12.711936 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.711914    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19502667-3b22-48ca-bb2c-bae2b3e3c391-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19502667-3b22-48ca-bb2c-bae2b3e3c391" (UID: "19502667-3b22-48ca-bb2c-bae2b3e3c391"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:41:12.712046 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.712031    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19502667-3b22-48ca-bb2c-bae2b3e3c391-kube-api-access-tdbxm" (OuterVolumeSpecName: "kube-api-access-tdbxm") pod "19502667-3b22-48ca-bb2c-bae2b3e3c391" (UID: "19502667-3b22-48ca-bb2c-bae2b3e3c391"). InnerVolumeSpecName "kube-api-access-tdbxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:41:12.719664 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.719640    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19502667-3b22-48ca-bb2c-bae2b3e3c391-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19502667-3b22-48ca-bb2c-bae2b3e3c391" (UID: "19502667-3b22-48ca-bb2c-bae2b3e3c391"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:41:12.811004 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.810977    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19502667-3b22-48ca-bb2c-bae2b3e3c391-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:41:12.811146 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.811022    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19502667-3b22-48ca-bb2c-bae2b3e3c391-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:41:12.811146 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.811049    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdbxm\" (UniqueName: \"kubernetes.io/projected/19502667-3b22-48ca-bb2c-bae2b3e3c391-kube-api-access-tdbxm\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:41:12.811146 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.811068    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19502667-3b22-48ca-bb2c-bae2b3e3c391-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:41:12.989064 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.989033    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"]
May 25 11:41:12.996400 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:12.996377    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8cghp"]
May 25 11:41:14.570007 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:14.569967    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" path="/var/lib/kubelet/pods/19502667-3b22-48ca-bb2c-bae2b3e3c391/volumes"
May 25 11:41:15.677284 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:15.677246    2569 generic.go:358] "Generic (PLEG): container finished" podID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerID="8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4" exitCode=0
May 25 11:41:15.677671 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:15.677322    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" event={"ID":"8f49bc95-7fe7-413f-b520-9c1ed31527d1","Type":"ContainerDied","Data":"8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4"}
May 25 11:41:16.681024 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:16.680994    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" event={"ID":"8f49bc95-7fe7-413f-b520-9c1ed31527d1","Type":"ContainerStarted","Data":"b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0"}
May 25 11:41:16.681393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:16.681032    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" event={"ID":"8f49bc95-7fe7-413f-b520-9c1ed31527d1","Type":"ContainerStarted","Data":"6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3"}
May 25 11:41:16.681393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:16.681242    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:16.727337 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:16.727296    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podStartSLOduration=6.727284657 podStartE2EDuration="6.727284657s" podCreationTimestamp="2026-05-25 11:41:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:41:16.726489662 +0000 UTC m=+1444.722664474" watchObservedRunningTime="2026-05-25 11:41:16.727284657 +0000 UTC m=+1444.723459471"
May 25 11:41:17.683717 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:17.683682    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:17.684715 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:17.684681    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 25 11:41:18.686508 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:18.686472    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 25 11:41:23.691276 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:23.691249    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:41:23.691874 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:23.691842    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 25 11:41:33.692699 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:33.692654    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 25 11:41:43.691754 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:43.691713    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 25 11:41:53.692280 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:41:53.692240    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 25 11:42:03.692835 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:03.692805    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:42:11.801617 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.801586    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"]
May 25 11:42:11.801992 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.801886    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container" containerID="cri-o://6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3" gracePeriod=30
May 25 11:42:11.801992 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.801918    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kube-rbac-proxy" containerID="cri-o://b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0" gracePeriod=30
May 25 11:42:11.926759 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.926731    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"]
May 25 11:42:11.927018 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.927005    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container"
May 25 11:42:11.927067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.927019    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container"
May 25 11:42:11.927067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.927032    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="storage-initializer"
May 25 11:42:11.927067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.927037    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="storage-initializer"
May 25 11:42:11.927067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.927044    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kube-rbac-proxy"
May 25 11:42:11.927067 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.927049    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kube-rbac-proxy"
May 25 11:42:11.927215 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.927092    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kube-rbac-proxy"
May 25 11:42:11.927215 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.927103    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="19502667-3b22-48ca-bb2c-bae2b3e3c391" containerName="kserve-container"
May 25 11:42:11.929917 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.929901    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:11.934328 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.934307    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
May 25 11:42:11.934520 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.934501    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
May 25 11:42:11.942436 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:11.942417    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"]
May 25 11:42:12.032217 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.032189    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.032368 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.032244    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gkw\" (UniqueName: \"kubernetes.io/projected/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kube-api-access-42gkw\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.032368 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.032270    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.032368 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.032296    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.133587 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.133506    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.133690 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.133604    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-42gkw\" (UniqueName: \"kubernetes.io/projected/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kube-api-access-42gkw\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.133690 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.133632    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.133690 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.133657    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.134082 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.134060    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.134265 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.134247    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.135948 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.135932    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.142558 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.142536    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gkw\" (UniqueName: \"kubernetes.io/projected/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kube-api-access-42gkw\") pod \"isvc-pmml-predictor-8bb578669-sfwbb\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.239543 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.239512    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:12.376070 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.376037    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"]
May 25 11:42:12.380123 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:42:12.380091    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb88f2a_376b_4df1_a0c8_6a0650cbfd79.slice/crio-eed059e7b50dfae67621b96b2a59c57a2450f56c6484e3dca369a0251878328f WatchSource:0}: Error finding container eed059e7b50dfae67621b96b2a59c57a2450f56c6484e3dca369a0251878328f: Status 404 returned error can't find the container with id eed059e7b50dfae67621b96b2a59c57a2450f56c6484e3dca369a0251878328f
May 25 11:42:12.381849 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.381832    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:42:12.840573 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.840534    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" event={"ID":"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79","Type":"ContainerStarted","Data":"322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b"}
May 25 11:42:12.841030 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.840577    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" event={"ID":"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79","Type":"ContainerStarted","Data":"eed059e7b50dfae67621b96b2a59c57a2450f56c6484e3dca369a0251878328f"}
May 25 11:42:12.842535 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.842506    2569 generic.go:358] "Generic (PLEG): container finished" podID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerID="b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0" exitCode=2
May 25 11:42:12.842671 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:12.842565    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" event={"ID":"8f49bc95-7fe7-413f-b520-9c1ed31527d1","Type":"ContainerDied","Data":"b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0"}
May 25 11:42:13.091853 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:13.091669    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:42:13.107869 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:13.095729    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:42:13.687420 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:13.687381    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 25 11:42:13.692701 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:13.692677    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 25 11:42:14.337882 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.337859    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:42:14.449724 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.449651    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kserve-provision-location\") pod \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") "
May 25 11:42:14.449724 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.449689    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f49bc95-7fe7-413f-b520-9c1ed31527d1-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") "
May 25 11:42:14.449926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.449729    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f49bc95-7fe7-413f-b520-9c1ed31527d1-proxy-tls\") pod \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") "
May 25 11:42:14.449926 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.449745    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm5kf\" (UniqueName: \"kubernetes.io/projected/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kube-api-access-xm5kf\") pod \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\" (UID: \"8f49bc95-7fe7-413f-b520-9c1ed31527d1\") "
May 25 11:42:14.450114 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.450079    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f49bc95-7fe7-413f-b520-9c1ed31527d1-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "8f49bc95-7fe7-413f-b520-9c1ed31527d1" (UID: "8f49bc95-7fe7-413f-b520-9c1ed31527d1"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:42:14.451764 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.451738    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f49bc95-7fe7-413f-b520-9c1ed31527d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f49bc95-7fe7-413f-b520-9c1ed31527d1" (UID: "8f49bc95-7fe7-413f-b520-9c1ed31527d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:42:14.451864 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.451806    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kube-api-access-xm5kf" (OuterVolumeSpecName: "kube-api-access-xm5kf") pod "8f49bc95-7fe7-413f-b520-9c1ed31527d1" (UID: "8f49bc95-7fe7-413f-b520-9c1ed31527d1"). InnerVolumeSpecName "kube-api-access-xm5kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:42:14.458418 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.458398    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f49bc95-7fe7-413f-b520-9c1ed31527d1" (UID: "8f49bc95-7fe7-413f-b520-9c1ed31527d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:42:14.550983 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.550958    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:42:14.550983 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.550981    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f49bc95-7fe7-413f-b520-9c1ed31527d1-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:42:14.551114 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.550993    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f49bc95-7fe7-413f-b520-9c1ed31527d1-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:42:14.551114 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.551004    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5kf\" (UniqueName: \"kubernetes.io/projected/8f49bc95-7fe7-413f-b520-9c1ed31527d1-kube-api-access-xm5kf\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:42:14.849608 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.849571    2569 generic.go:358] "Generic (PLEG): container finished" podID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerID="6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3" exitCode=0
May 25 11:42:14.849905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.849636    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" event={"ID":"8f49bc95-7fe7-413f-b520-9c1ed31527d1","Type":"ContainerDied","Data":"6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3"}
May 25 11:42:14.849905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.849673    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8" event={"ID":"8f49bc95-7fe7-413f-b520-9c1ed31527d1","Type":"ContainerDied","Data":"2a2a41e759a11a2cada7f357650ac72b90058466e5cd7039aa61b52f466ebd4b"}
May 25 11:42:14.849905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.849675    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"
May 25 11:42:14.849905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.849693    2569 scope.go:117] "RemoveContainer" containerID="b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0"
May 25 11:42:14.857194 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.857178    2569 scope.go:117] "RemoveContainer" containerID="6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3"
May 25 11:42:14.863894 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.863877    2569 scope.go:117] "RemoveContainer" containerID="8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4"
May 25 11:42:14.868894 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.868873    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"]
May 25 11:42:14.870702 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.870686    2569 scope.go:117] "RemoveContainer" containerID="b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0"
May 25 11:42:14.870974 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:42:14.870956    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0\": container with ID starting with b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0 not found: ID does not exist" containerID="b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0"
May 25 11:42:14.871035 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.870982    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0"} err="failed to get container status \"b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0\": rpc error: code = NotFound desc = could not find container \"b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0\": container with ID starting with b3b6f7947a16b712d4b4693313e7c567f7aa0636f38515d79c8e24d2f156f9d0 not found: ID does not exist"
May 25 11:42:14.871035 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.871000    2569 scope.go:117] "RemoveContainer" containerID="6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3"
May 25 11:42:14.871239 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:42:14.871225    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3\": container with ID starting with 6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3 not found: ID does not exist" containerID="6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3"
May 25 11:42:14.871291 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.871241    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3"} err="failed to get container status \"6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3\": rpc error: code = NotFound desc = could not find container \"6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3\": container with ID starting with 6c9f0e03502b7044f1bff2be5e5faea3f5d13a6e5549f9d3f9fa30a5430667e3 not found: ID does not exist"
May 25 11:42:14.871291 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.871253    2569 scope.go:117] "RemoveContainer" containerID="8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4"
May 25 11:42:14.871466 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:42:14.871450    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4\": container with ID starting with 8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4 not found: ID does not exist" containerID="8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4"
May 25 11:42:14.871506 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.871471    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4"} err="failed to get container status \"8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4\": rpc error: code = NotFound desc = could not find container \"8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4\": container with ID starting with 8918f5991a002eee43f98368c6dad132759360ce80ad7eb394bf3900d54af5a4 not found: ID does not exist"
May 25 11:42:14.875347 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:14.875324    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-nx6f8"]
May 25 11:42:16.571089 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:16.571050    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" path="/var/lib/kubelet/pods/8f49bc95-7fe7-413f-b520-9c1ed31527d1/volumes"
May 25 11:42:16.857337 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:16.857256    2569 generic.go:358] "Generic (PLEG): container finished" podID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerID="322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b" exitCode=0
May 25 11:42:16.857475 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:16.857331    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" event={"ID":"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79","Type":"ContainerDied","Data":"322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b"}
May 25 11:42:23.888352 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:23.886288    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" event={"ID":"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79","Type":"ContainerStarted","Data":"94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f"}
May 25 11:42:23.888352 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:23.886330    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" event={"ID":"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79","Type":"ContainerStarted","Data":"e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546"}
May 25 11:42:23.888352 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:23.888343    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:23.888907 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:23.888376    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:23.888907 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:23.888546    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 25 11:42:23.905628 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:23.905586    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podStartSLOduration=6.12722074 podStartE2EDuration="12.905574723s" podCreationTimestamp="2026-05-25 11:42:11 +0000 UTC" firstStartedPulling="2026-05-25 11:42:16.858439031 +0000 UTC m=+1504.854613821" lastFinishedPulling="2026-05-25 11:42:23.636792994 +0000 UTC m=+1511.632967804" observedRunningTime="2026-05-25 11:42:23.904250512 +0000 UTC m=+1511.900425326" watchObservedRunningTime="2026-05-25 11:42:23.905574723 +0000 UTC m=+1511.901749535"
May 25 11:42:24.889439 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:24.889350    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 25 11:42:29.893970 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:29.893943    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:42:29.894522 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:29.894492    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 25 11:42:39.895364 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:39.895332    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 25 11:42:49.895357 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:49.895316    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 25 11:42:59.894837 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:42:59.894788    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 25 11:43:09.894480 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:09.894442    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 25 11:43:19.895256 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:19.895211    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 25 11:43:29.895125 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:29.895087    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 25 11:43:39.895247 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:39.895218    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:43:43.518550 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.518519    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"]
May 25 11:43:43.518967 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.518835    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container" containerID="cri-o://e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546" gracePeriod=30
May 25 11:43:43.518967 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.518888    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kube-rbac-proxy" containerID="cri-o://94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f" gracePeriod=30
May 25 11:43:43.669764 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.669729    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"]
May 25 11:43:43.670032 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.670019    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="storage-initializer"
May 25 11:43:43.670077 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.670033    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="storage-initializer"
May 25 11:43:43.670077 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.670043    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container"
May 25 11:43:43.670077 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.670048    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container"
May 25 11:43:43.670077 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.670057    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kube-rbac-proxy"
May 25 11:43:43.670077 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.670062    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kube-rbac-proxy"
May 25 11:43:43.670229 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.670128    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kserve-container"
May 25 11:43:43.670229 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.670138    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f49bc95-7fe7-413f-b520-9c1ed31527d1" containerName="kube-rbac-proxy"
May 25 11:43:43.677801 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.677758    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.680177 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.680159    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
May 25 11:43:43.680287 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.680160    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
May 25 11:43:43.683147 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.683112    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"]
May 25 11:43:43.707654 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.707633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66f9995-8876-47f1-a25b-d3ab53ff9917-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.707744 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.707679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66f9995-8876-47f1-a25b-d3ab53ff9917-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.707744 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.707701    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbt8k\" (UniqueName: \"kubernetes.io/projected/e66f9995-8876-47f1-a25b-d3ab53ff9917-kube-api-access-sbt8k\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.707838 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.707763    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66f9995-8876-47f1-a25b-d3ab53ff9917-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.809060 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.808989    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66f9995-8876-47f1-a25b-d3ab53ff9917-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.809060 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.809025    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbt8k\" (UniqueName: \"kubernetes.io/projected/e66f9995-8876-47f1-a25b-d3ab53ff9917-kube-api-access-sbt8k\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.809060 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.809047    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66f9995-8876-47f1-a25b-d3ab53ff9917-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.809293 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.809084    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66f9995-8876-47f1-a25b-d3ab53ff9917-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.809293 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:43:43.809145    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
May 25 11:43:43.809293 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:43:43.809213    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e66f9995-8876-47f1-a25b-d3ab53ff9917-proxy-tls podName:e66f9995-8876-47f1-a25b-d3ab53ff9917 nodeName:}" failed. No retries permitted until 2026-05-25 11:43:44.309185172 +0000 UTC m=+1592.305359963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e66f9995-8876-47f1-a25b-d3ab53ff9917-proxy-tls") pod "isvc-pmml-runtime-predictor-67bc544947-tgx7q" (UID: "e66f9995-8876-47f1-a25b-d3ab53ff9917") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
May 25 11:43:43.809472 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.809453    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66f9995-8876-47f1-a25b-d3ab53ff9917-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.809791 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.809755    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66f9995-8876-47f1-a25b-d3ab53ff9917-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:43.817441 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:43.817423    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbt8k\" (UniqueName: \"kubernetes.io/projected/e66f9995-8876-47f1-a25b-d3ab53ff9917-kube-api-access-sbt8k\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:44.123402 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:44.123319    2569 generic.go:358] "Generic (PLEG): container finished" podID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerID="94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f" exitCode=2
May 25 11:43:44.123402 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:44.123362    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" event={"ID":"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79","Type":"ContainerDied","Data":"94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f"}
May 25 11:43:44.313115 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:44.313073    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66f9995-8876-47f1-a25b-d3ab53ff9917-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:44.315380 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:44.315359    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66f9995-8876-47f1-a25b-d3ab53ff9917-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-tgx7q\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:44.588055 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:44.588024    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:44.712490 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:44.712464    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"]
May 25 11:43:44.714183 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:43:44.714158    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66f9995_8876_47f1_a25b_d3ab53ff9917.slice/crio-428a2016cb88dc8efb8b9120c3510a4fd89445d7e918d6d6afc059884a8c3318 WatchSource:0}: Error finding container 428a2016cb88dc8efb8b9120c3510a4fd89445d7e918d6d6afc059884a8c3318: Status 404 returned error can't find the container with id 428a2016cb88dc8efb8b9120c3510a4fd89445d7e918d6d6afc059884a8c3318
May 25 11:43:44.890680 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:44.890573    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
May 25 11:43:45.127491 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:45.127455    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" event={"ID":"e66f9995-8876-47f1-a25b-d3ab53ff9917","Type":"ContainerStarted","Data":"981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d"}
May 25 11:43:45.127491 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:45.127494    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" event={"ID":"e66f9995-8876-47f1-a25b-d3ab53ff9917","Type":"ContainerStarted","Data":"428a2016cb88dc8efb8b9120c3510a4fd89445d7e918d6d6afc059884a8c3318"}
May 25 11:43:46.954972 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:46.954947    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:43:47.034050 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.033974    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42gkw\" (UniqueName: \"kubernetes.io/projected/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kube-api-access-42gkw\") pod \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") "
May 25 11:43:47.034050 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.034011    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kserve-provision-location\") pod \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") "
May 25 11:43:47.034050 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.034036    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") "
May 25 11:43:47.034253 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.034081    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-proxy-tls\") pod \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\" (UID: \"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79\") "
May 25 11:43:47.034349 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.034326    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" (UID: "fbb88f2a-376b-4df1-a0c8-6a0650cbfd79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:47.034411 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.034392    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" (UID: "fbb88f2a-376b-4df1-a0c8-6a0650cbfd79"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:43:47.036088 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.036062    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" (UID: "fbb88f2a-376b-4df1-a0c8-6a0650cbfd79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:43:47.036145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.036089    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kube-api-access-42gkw" (OuterVolumeSpecName: "kube-api-access-42gkw") pod "fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" (UID: "fbb88f2a-376b-4df1-a0c8-6a0650cbfd79"). InnerVolumeSpecName "kube-api-access-42gkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:43:47.134504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.134478    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:43:47.134504 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.134502    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gkw\" (UniqueName: \"kubernetes.io/projected/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kube-api-access-42gkw\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:43:47.134698 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.134516    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:43:47.134698 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.134529    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:43:47.135044 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.135021    2569 generic.go:358] "Generic (PLEG): container finished" podID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerID="e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546" exitCode=0
May 25 11:43:47.135111 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.135097    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"
May 25 11:43:47.135111 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.135099    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" event={"ID":"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79","Type":"ContainerDied","Data":"e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546"}
May 25 11:43:47.135184 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.135137    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb" event={"ID":"fbb88f2a-376b-4df1-a0c8-6a0650cbfd79","Type":"ContainerDied","Data":"eed059e7b50dfae67621b96b2a59c57a2450f56c6484e3dca369a0251878328f"}
May 25 11:43:47.135184 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.135153    2569 scope.go:117] "RemoveContainer" containerID="94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f"
May 25 11:43:47.144759 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.144740    2569 scope.go:117] "RemoveContainer" containerID="e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546"
May 25 11:43:47.151678 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.151663    2569 scope.go:117] "RemoveContainer" containerID="322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b"
May 25 11:43:47.157842 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.157825    2569 scope.go:117] "RemoveContainer" containerID="94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f"
May 25 11:43:47.158084 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:43:47.158066    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f\": container with ID starting with 94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f not found: ID does not exist" containerID="94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f"
May 25 11:43:47.158150 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.158096    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f"} err="failed to get container status \"94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f\": rpc error: code = NotFound desc = could not find container \"94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f\": container with ID starting with 94a50c92de0e60a5725e45524d4f6329b2afd4251dea07be8e86ea533172341f not found: ID does not exist"
May 25 11:43:47.158150 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.158121    2569 scope.go:117] "RemoveContainer" containerID="e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546"
May 25 11:43:47.158359 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:43:47.158343    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546\": container with ID starting with e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546 not found: ID does not exist" containerID="e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546"
May 25 11:43:47.158397 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.158366    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546"} err="failed to get container status \"e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546\": rpc error: code = NotFound desc = could not find container \"e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546\": container with ID starting with e37512887039beac15f830c29488e2b7fd624d0cc27b42d2752271455d864546 not found: ID does not exist"
May 25 11:43:47.158397 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.158380    2569 scope.go:117] "RemoveContainer" containerID="322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b"
May 25 11:43:47.158605 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:43:47.158589    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b\": container with ID starting with 322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b not found: ID does not exist" containerID="322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b"
May 25 11:43:47.158657 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.158610    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b"} err="failed to get container status \"322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b\": rpc error: code = NotFound desc = could not find container \"322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b\": container with ID starting with 322d7ceb0ffc10cf3141db4a8b4ab6747a5f95021f7ffd7b4a1d664e3a900c8b not found: ID does not exist"
May 25 11:43:47.161179 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.161159    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"]
May 25 11:43:47.182226 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:47.182207    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sfwbb"]
May 25 11:43:48.569439 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:48.569404    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" path="/var/lib/kubelet/pods/fbb88f2a-376b-4df1-a0c8-6a0650cbfd79/volumes"
May 25 11:43:49.143162 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:49.143124    2569 generic.go:358] "Generic (PLEG): container finished" podID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerID="981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d" exitCode=0
May 25 11:43:49.143328 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:49.143172    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" event={"ID":"e66f9995-8876-47f1-a25b-d3ab53ff9917","Type":"ContainerDied","Data":"981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d"}
May 25 11:43:50.148221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:50.148186    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" event={"ID":"e66f9995-8876-47f1-a25b-d3ab53ff9917","Type":"ContainerStarted","Data":"57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f"}
May 25 11:43:50.148221 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:50.148222    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" event={"ID":"e66f9995-8876-47f1-a25b-d3ab53ff9917","Type":"ContainerStarted","Data":"78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca"}
May 25 11:43:50.148657 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:50.148504    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:50.148657 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:50.148614    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:50.149868 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:50.149836    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:43:50.169989 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:50.169945    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podStartSLOduration=7.169933681 podStartE2EDuration="7.169933681s" podCreationTimestamp="2026-05-25 11:43:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:43:50.168289785 +0000 UTC m=+1598.164464595" watchObservedRunningTime="2026-05-25 11:43:50.169933681 +0000 UTC m=+1598.166108494"
May 25 11:43:51.151476 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:51.151437    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:43:56.157207 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:56.157133    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:43:56.157684 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:43:56.157650    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:44:06.158534 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:44:06.158497    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:44:16.158192 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:44:16.158152    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:44:26.158413 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:44:26.158375    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:44:36.157928 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:44:36.157887    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:44:46.157781 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:44:46.157730    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:44:56.158469 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:44:56.158426    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:45:06.158485 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:06.158441    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 25 11:45:09.566891 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:09.566860    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:45:14.692960 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.692924    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"]
May 25 11:45:14.693355 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.693330    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container" containerID="cri-o://78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca" gracePeriod=30
May 25 11:45:14.693413 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.693362    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kube-rbac-proxy" containerID="cri-o://57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f" gracePeriod=30
May 25 11:45:14.843703 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.843668    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"]
May 25 11:45:14.843982 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.843966    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kube-rbac-proxy"
May 25 11:45:14.843982 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.843982    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kube-rbac-proxy"
May 25 11:45:14.844123 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.843994    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="storage-initializer"
May 25 11:45:14.844123 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.844000    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="storage-initializer"
May 25 11:45:14.844123 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.844012    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container"
May 25 11:45:14.844123 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.844018    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container"
May 25 11:45:14.844123 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.844065    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kube-rbac-proxy"
May 25 11:45:14.844123 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.844074    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbb88f2a-376b-4df1-a0c8-6a0650cbfd79" containerName="kserve-container"
May 25 11:45:14.846949 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.846929    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:14.849474 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.849451    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
May 25 11:45:14.850104 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.850079    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
May 25 11:45:14.860835 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.860814    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"]
May 25 11:45:14.962571 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.962464    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51a5bcb2-a205-40e0-a033-2a0c66a04728-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:14.962571 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.962528    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f8s6\" (UniqueName: \"kubernetes.io/projected/51a5bcb2-a205-40e0-a033-2a0c66a04728-kube-api-access-5f8s6\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:14.962571 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.962557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51a5bcb2-a205-40e0-a033-2a0c66a04728-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:14.962830 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:14.962602    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51a5bcb2-a205-40e0-a033-2a0c66a04728-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.063357 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.063318    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51a5bcb2-a205-40e0-a033-2a0c66a04728-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.063357 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.063364    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5f8s6\" (UniqueName: \"kubernetes.io/projected/51a5bcb2-a205-40e0-a033-2a0c66a04728-kube-api-access-5f8s6\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.063699 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.063391    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51a5bcb2-a205-40e0-a033-2a0c66a04728-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.063699 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.063424    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51a5bcb2-a205-40e0-a033-2a0c66a04728-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.063851 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.063816    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51a5bcb2-a205-40e0-a033-2a0c66a04728-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.064072 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.064053    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51a5bcb2-a205-40e0-a033-2a0c66a04728-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.065879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.065859    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51a5bcb2-a205-40e0-a033-2a0c66a04728-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.071481 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.071458    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f8s6\" (UniqueName: \"kubernetes.io/projected/51a5bcb2-a205-40e0-a033-2a0c66a04728-kube-api-access-5f8s6\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.156301 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.156264    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:15.281036 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.280997    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"]
May 25 11:45:15.283140 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:45:15.283103    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a5bcb2_a205_40e0_a033_2a0c66a04728.slice/crio-31d219169ef0be92220f6e04d3b72d3c83a9a08d29aa88796c22e0303c3870cf WatchSource:0}: Error finding container 31d219169ef0be92220f6e04d3b72d3c83a9a08d29aa88796c22e0303c3870cf: Status 404 returned error can't find the container with id 31d219169ef0be92220f6e04d3b72d3c83a9a08d29aa88796c22e0303c3870cf
May 25 11:45:15.381945 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.381905    2569 generic.go:358] "Generic (PLEG): container finished" podID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerID="57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f" exitCode=2
May 25 11:45:15.382098 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.381967    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" event={"ID":"e66f9995-8876-47f1-a25b-d3ab53ff9917","Type":"ContainerDied","Data":"57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f"}
May 25 11:45:15.383329 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.383308    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" event={"ID":"51a5bcb2-a205-40e0-a033-2a0c66a04728","Type":"ContainerStarted","Data":"5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004"}
May 25 11:45:15.383429 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:15.383333    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" event={"ID":"51a5bcb2-a205-40e0-a033-2a0c66a04728","Type":"ContainerStarted","Data":"31d219169ef0be92220f6e04d3b72d3c83a9a08d29aa88796c22e0303c3870cf"}
May 25 11:45:16.152358 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:16.152322    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
May 25 11:45:18.033422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.033400    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:45:18.088386 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.088360    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66f9995-8876-47f1-a25b-d3ab53ff9917-kserve-provision-location\") pod \"e66f9995-8876-47f1-a25b-d3ab53ff9917\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") "
May 25 11:45:18.088561 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.088418    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbt8k\" (UniqueName: \"kubernetes.io/projected/e66f9995-8876-47f1-a25b-d3ab53ff9917-kube-api-access-sbt8k\") pod \"e66f9995-8876-47f1-a25b-d3ab53ff9917\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") "
May 25 11:45:18.088561 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.088450    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66f9995-8876-47f1-a25b-d3ab53ff9917-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"e66f9995-8876-47f1-a25b-d3ab53ff9917\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") "
May 25 11:45:18.088561 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.088477    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66f9995-8876-47f1-a25b-d3ab53ff9917-proxy-tls\") pod \"e66f9995-8876-47f1-a25b-d3ab53ff9917\" (UID: \"e66f9995-8876-47f1-a25b-d3ab53ff9917\") "
May 25 11:45:18.088736 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.088703    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66f9995-8876-47f1-a25b-d3ab53ff9917-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e66f9995-8876-47f1-a25b-d3ab53ff9917" (UID: "e66f9995-8876-47f1-a25b-d3ab53ff9917"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:45:18.088864 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.088839    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66f9995-8876-47f1-a25b-d3ab53ff9917-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "e66f9995-8876-47f1-a25b-d3ab53ff9917" (UID: "e66f9995-8876-47f1-a25b-d3ab53ff9917"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:45:18.090549 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.090531    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66f9995-8876-47f1-a25b-d3ab53ff9917-kube-api-access-sbt8k" (OuterVolumeSpecName: "kube-api-access-sbt8k") pod "e66f9995-8876-47f1-a25b-d3ab53ff9917" (UID: "e66f9995-8876-47f1-a25b-d3ab53ff9917"). InnerVolumeSpecName "kube-api-access-sbt8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:45:18.090610 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.090552    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66f9995-8876-47f1-a25b-d3ab53ff9917-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e66f9995-8876-47f1-a25b-d3ab53ff9917" (UID: "e66f9995-8876-47f1-a25b-d3ab53ff9917"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:45:18.189447 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.189382    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e66f9995-8876-47f1-a25b-d3ab53ff9917-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:45:18.189447 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.189405    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e66f9995-8876-47f1-a25b-d3ab53ff9917-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:45:18.189447 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.189415    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66f9995-8876-47f1-a25b-d3ab53ff9917-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:45:18.189447 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.189426    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbt8k\" (UniqueName: \"kubernetes.io/projected/e66f9995-8876-47f1-a25b-d3ab53ff9917-kube-api-access-sbt8k\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:45:18.393982 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.393951    2569 generic.go:358] "Generic (PLEG): container finished" podID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerID="78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca" exitCode=0
May 25 11:45:18.394141 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.394031    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" event={"ID":"e66f9995-8876-47f1-a25b-d3ab53ff9917","Type":"ContainerDied","Data":"78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca"}
May 25 11:45:18.394141 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.394048    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"
May 25 11:45:18.394141 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.394070    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q" event={"ID":"e66f9995-8876-47f1-a25b-d3ab53ff9917","Type":"ContainerDied","Data":"428a2016cb88dc8efb8b9120c3510a4fd89445d7e918d6d6afc059884a8c3318"}
May 25 11:45:18.394141 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.394086    2569 scope.go:117] "RemoveContainer" containerID="57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f"
May 25 11:45:18.401891 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.401874    2569 scope.go:117] "RemoveContainer" containerID="78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca"
May 25 11:45:18.409143 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.409126    2569 scope.go:117] "RemoveContainer" containerID="981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d"
May 25 11:45:18.415897 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.415867    2569 scope.go:117] "RemoveContainer" containerID="57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f"
May 25 11:45:18.416204 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:45:18.416177    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f\": container with ID starting with 57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f not found: ID does not exist" containerID="57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f"
May 25 11:45:18.416285 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.416213    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f"} err="failed to get container status \"57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f\": rpc error: code = NotFound desc = could not find container \"57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f\": container with ID starting with 57d779b8cdde14498f0e35ef748881b9bafd45a52d72844f258147cfde80062f not found: ID does not exist"
May 25 11:45:18.416285 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.416235    2569 scope.go:117] "RemoveContainer" containerID="78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca"
May 25 11:45:18.416544 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:45:18.416520    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca\": container with ID starting with 78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca not found: ID does not exist" containerID="78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca"
May 25 11:45:18.416592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.416563    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca"} err="failed to get container status \"78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca\": rpc error: code = NotFound desc = could not find container \"78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca\": container with ID starting with 78ab57c0ab8c8046e785c50f30260cb309581f42b22d8ee3bdfcf37072f128ca not found: ID does not exist"
May 25 11:45:18.416592 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.416585    2569 scope.go:117] "RemoveContainer" containerID="981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d"
May 25 11:45:18.416841 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:45:18.416816    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d\": container with ID starting with 981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d not found: ID does not exist" containerID="981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d"
May 25 11:45:18.416905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.416851    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d"} err="failed to get container status \"981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d\": rpc error: code = NotFound desc = could not find container \"981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d\": container with ID starting with 981bd6cdefedb7cc9296d0870ab775f5d5093f67cbc347f1920356bc209bf36d not found: ID does not exist"
May 25 11:45:18.417051 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.417033    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"]
May 25 11:45:18.418934 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.418914    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-tgx7q"]
May 25 11:45:18.568907 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:18.568879    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" path="/var/lib/kubelet/pods/e66f9995-8876-47f1-a25b-d3ab53ff9917/volumes"
May 25 11:45:19.402735 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:19.402638    2569 generic.go:358] "Generic (PLEG): container finished" podID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerID="5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004" exitCode=0
May 25 11:45:19.402735 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:19.402711    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" event={"ID":"51a5bcb2-a205-40e0-a033-2a0c66a04728","Type":"ContainerDied","Data":"5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004"}
May 25 11:45:20.408102 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:20.408069    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" event={"ID":"51a5bcb2-a205-40e0-a033-2a0c66a04728","Type":"ContainerStarted","Data":"f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4"}
May 25 11:45:20.408466 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:20.408109    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" event={"ID":"51a5bcb2-a205-40e0-a033-2a0c66a04728","Type":"ContainerStarted","Data":"7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0"}
May 25 11:45:20.408466 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:20.408395    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:20.408538 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:20.408505    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:20.409904 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:20.409871    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:45:20.427646 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:20.427605    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podStartSLOduration=6.427593427 podStartE2EDuration="6.427593427s" podCreationTimestamp="2026-05-25 11:45:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:45:20.425895691 +0000 UTC m=+1688.422070514" watchObservedRunningTime="2026-05-25 11:45:20.427593427 +0000 UTC m=+1688.423768239"
May 25 11:45:21.411416 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:21.411374    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:45:26.416321 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:26.416240    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:45:26.416710 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:26.416687    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:45:36.417563 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:36.417523    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:45:46.417294 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:46.417255    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:45:56.417233 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:45:56.417184    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:46:06.417327 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:06.417290    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:46:16.417488 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:16.417442    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:46:26.417061 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:26.417026    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:46:36.417368 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:36.417326    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:46:46.416922 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:46.416885    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:46:56.089099 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:56.089024    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"]
May 25 11:46:56.089567 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:56.089369    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" containerID="cri-o://7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0" gracePeriod=30
May 25 11:46:56.089567 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:56.089413    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kube-rbac-proxy" containerID="cri-o://f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4" gracePeriod=30
May 25 11:46:56.412441 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:56.412354    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
May 25 11:46:56.416661 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:56.416640    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 25 11:46:56.686762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:56.686677    2569 generic.go:358] "Generic (PLEG): container finished" podID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerID="f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4" exitCode=2
May 25 11:46:56.686762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:56.686724    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" event={"ID":"51a5bcb2-a205-40e0-a033-2a0c66a04728","Type":"ContainerDied","Data":"f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4"}
May 25 11:46:59.532910 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.532888    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:46:59.611191 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.611115    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51a5bcb2-a205-40e0-a033-2a0c66a04728-kserve-provision-location\") pod \"51a5bcb2-a205-40e0-a033-2a0c66a04728\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") "
May 25 11:46:59.611191 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.611165    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51a5bcb2-a205-40e0-a033-2a0c66a04728-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"51a5bcb2-a205-40e0-a033-2a0c66a04728\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") "
May 25 11:46:59.611360 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.611236    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51a5bcb2-a205-40e0-a033-2a0c66a04728-proxy-tls\") pod \"51a5bcb2-a205-40e0-a033-2a0c66a04728\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") "
May 25 11:46:59.611360 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.611261    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f8s6\" (UniqueName: \"kubernetes.io/projected/51a5bcb2-a205-40e0-a033-2a0c66a04728-kube-api-access-5f8s6\") pod \"51a5bcb2-a205-40e0-a033-2a0c66a04728\" (UID: \"51a5bcb2-a205-40e0-a033-2a0c66a04728\") "
May 25 11:46:59.611497 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.611476    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a5bcb2-a205-40e0-a033-2a0c66a04728-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51a5bcb2-a205-40e0-a033-2a0c66a04728" (UID: "51a5bcb2-a205-40e0-a033-2a0c66a04728"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:59.611598 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.611572    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a5bcb2-a205-40e0-a033-2a0c66a04728-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "51a5bcb2-a205-40e0-a033-2a0c66a04728" (UID: "51a5bcb2-a205-40e0-a033-2a0c66a04728"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:46:59.613259 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.613230    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a5bcb2-a205-40e0-a033-2a0c66a04728-kube-api-access-5f8s6" (OuterVolumeSpecName: "kube-api-access-5f8s6") pod "51a5bcb2-a205-40e0-a033-2a0c66a04728" (UID: "51a5bcb2-a205-40e0-a033-2a0c66a04728"). InnerVolumeSpecName "kube-api-access-5f8s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:46:59.613559 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.613543    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a5bcb2-a205-40e0-a033-2a0c66a04728-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51a5bcb2-a205-40e0-a033-2a0c66a04728" (UID: "51a5bcb2-a205-40e0-a033-2a0c66a04728"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:46:59.696988 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.696952    2569 generic.go:358] "Generic (PLEG): container finished" podID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerID="7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0" exitCode=0
May 25 11:46:59.697170 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.697040    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"
May 25 11:46:59.697170 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.697038    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" event={"ID":"51a5bcb2-a205-40e0-a033-2a0c66a04728","Type":"ContainerDied","Data":"7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0"}
May 25 11:46:59.697170 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.697143    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp" event={"ID":"51a5bcb2-a205-40e0-a033-2a0c66a04728","Type":"ContainerDied","Data":"31d219169ef0be92220f6e04d3b72d3c83a9a08d29aa88796c22e0303c3870cf"}
May 25 11:46:59.697170 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.697168    2569 scope.go:117] "RemoveContainer" containerID="f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4"
May 25 11:46:59.704495 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.704477    2569 scope.go:117] "RemoveContainer" containerID="7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0"
May 25 11:46:59.711499 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.711483    2569 scope.go:117] "RemoveContainer" containerID="5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004"
May 25 11:46:59.711734 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.711714    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51a5bcb2-a205-40e0-a033-2a0c66a04728-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:46:59.711870 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.711738    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51a5bcb2-a205-40e0-a033-2a0c66a04728-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:46:59.711870 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.711748    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51a5bcb2-a205-40e0-a033-2a0c66a04728-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:46:59.711870 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.711758    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f8s6\" (UniqueName: \"kubernetes.io/projected/51a5bcb2-a205-40e0-a033-2a0c66a04728-kube-api-access-5f8s6\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:46:59.718284 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.718263    2569 scope.go:117] "RemoveContainer" containerID="f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4"
May 25 11:46:59.718551 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:46:59.718533    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4\": container with ID starting with f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4 not found: ID does not exist" containerID="f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4"
May 25 11:46:59.718634 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.718558    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4"} err="failed to get container status \"f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4\": rpc error: code = NotFound desc = could not find container \"f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4\": container with ID starting with f9e6ff25f6453f63227bb42ee4914c75b645ebec1d68071962b239c0b96a5ab4 not found: ID does not exist"
May 25 11:46:59.718634 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.718577    2569 scope.go:117] "RemoveContainer" containerID="7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0"
May 25 11:46:59.718848 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:46:59.718820    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0\": container with ID starting with 7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0 not found: ID does not exist" containerID="7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0"
May 25 11:46:59.718940 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.718853    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0"} err="failed to get container status \"7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0\": rpc error: code = NotFound desc = could not find container \"7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0\": container with ID starting with 7bc9fbe226b542a8e3d96103b2f8d93f857e9f62bfe8908c9039e1c2c12ee4e0 not found: ID does not exist"
May 25 11:46:59.718940 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.718876    2569 scope.go:117] "RemoveContainer" containerID="5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004"
May 25 11:46:59.719122 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:46:59.719102    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004\": container with ID starting with 5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004 not found: ID does not exist" containerID="5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004"
May 25 11:46:59.719166 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.719128    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004"} err="failed to get container status \"5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004\": rpc error: code = NotFound desc = could not find container \"5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004\": container with ID starting with 5009c2a70441829b9981fa2b85250a9b99ac38b05016ad15363c52ad55a3a004 not found: ID does not exist"
May 25 11:46:59.719388 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.719373    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"]
May 25 11:46:59.722931 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:46:59.722912    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vkblp"]
May 25 11:47:00.569114 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:47:00.569038    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" path="/var/lib/kubelet/pods/51a5bcb2-a205-40e0-a033-2a0c66a04728/volumes"
May 25 11:47:13.113875 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:47:13.113739    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:47:13.131380 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:47:13.118412    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:48:42.594387 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594308    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"]
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594583    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kube-rbac-proxy"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594595    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kube-rbac-proxy"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594604    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594609    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594618    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594623    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594631    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="storage-initializer"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594636    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="storage-initializer"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594641    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="storage-initializer"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594646    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="storage-initializer"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594655    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kube-rbac-proxy"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594660    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kube-rbac-proxy"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594700    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kube-rbac-proxy"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594708    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="e66f9995-8876-47f1-a25b-d3ab53ff9917" containerName="kserve-container"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594714    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kserve-container"
May 25 11:48:42.594879 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.594721    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="51a5bcb2-a205-40e0-a033-2a0c66a04728" containerName="kube-rbac-proxy"
May 25 11:48:42.602440 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.602418    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.605058 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.605031    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
May 25 11:48:42.605193 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.605061    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
May 25 11:48:42.605193 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.605080    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:48:42.605193 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.605097    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:48:42.605193 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.605085    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 11:48:42.608278 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.608258    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"]
May 25 11:48:42.619806 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.619764    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c558365f-3916-415b-b626-00da1e7d645a-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.619905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.619819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6pw9\" (UniqueName: \"kubernetes.io/projected/c558365f-3916-415b-b626-00da1e7d645a-kube-api-access-w6pw9\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.619905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.619847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c558365f-3916-415b-b626-00da1e7d645a-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.619905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.619863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558365f-3916-415b-b626-00da1e7d645a-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.720300 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.720270    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6pw9\" (UniqueName: \"kubernetes.io/projected/c558365f-3916-415b-b626-00da1e7d645a-kube-api-access-w6pw9\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.720442 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.720321    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c558365f-3916-415b-b626-00da1e7d645a-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.720442 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.720340    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558365f-3916-415b-b626-00da1e7d645a-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.720442 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.720393    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c558365f-3916-415b-b626-00da1e7d645a-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.720757 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.720733    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558365f-3916-415b-b626-00da1e7d645a-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.721055 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.721036    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c558365f-3916-415b-b626-00da1e7d645a-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.722947 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.722923    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c558365f-3916-415b-b626-00da1e7d645a-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.729076 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.729059    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6pw9\" (UniqueName: \"kubernetes.io/projected/c558365f-3916-415b-b626-00da1e7d645a-kube-api-access-w6pw9\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:42.913168 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:42.913081    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:48:43.037301 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:43.037276    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"]
May 25 11:48:43.039291 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:48:43.039263    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc558365f_3916_415b_b626_00da1e7d645a.slice/crio-59c035bb56060f0e77c810e657c9e0f01481a18c04f8c8ca5f5e7f5d10e2cf26 WatchSource:0}: Error finding container 59c035bb56060f0e77c810e657c9e0f01481a18c04f8c8ca5f5e7f5d10e2cf26: Status 404 returned error can't find the container with id 59c035bb56060f0e77c810e657c9e0f01481a18c04f8c8ca5f5e7f5d10e2cf26
May 25 11:48:43.041151 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:43.041117    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:48:43.975340 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:43.975301    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" event={"ID":"c558365f-3916-415b-b626-00da1e7d645a","Type":"ContainerStarted","Data":"ff19c21eaab49879715c58ea751b7297ae5328b744533ef2328a92f44906b56e"}
May 25 11:48:43.975340 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:43.975344    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" event={"ID":"c558365f-3916-415b-b626-00da1e7d645a","Type":"ContainerStarted","Data":"59c035bb56060f0e77c810e657c9e0f01481a18c04f8c8ca5f5e7f5d10e2cf26"}
May 25 11:48:46.985507 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:46.985472    2569 generic.go:358] "Generic (PLEG): container finished" podID="c558365f-3916-415b-b626-00da1e7d645a" containerID="ff19c21eaab49879715c58ea751b7297ae5328b744533ef2328a92f44906b56e" exitCode=0
May 25 11:48:46.985876 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:48:46.985520    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" event={"ID":"c558365f-3916-415b-b626-00da1e7d645a","Type":"ContainerDied","Data":"ff19c21eaab49879715c58ea751b7297ae5328b744533ef2328a92f44906b56e"}
May 25 11:49:07.056611 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:07.056574    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" event={"ID":"c558365f-3916-415b-b626-00da1e7d645a","Type":"ContainerStarted","Data":"142fb5f7a2735aec35179a16ec11ecc678d16a729facfc77f33abe8250faad04"}
May 25 11:49:07.056611 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:07.056614    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" event={"ID":"c558365f-3916-415b-b626-00da1e7d645a","Type":"ContainerStarted","Data":"d21c6bd54a4bf5243d21b77da020ef8d3eb0326e3ca69a7ae8149fd8b0906695"}
May 25 11:49:07.057041 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:07.056805    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:49:07.076445 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:07.076402    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podStartSLOduration=5.352457039 podStartE2EDuration="25.076389819s" podCreationTimestamp="2026-05-25 11:48:42 +0000 UTC" firstStartedPulling="2026-05-25 11:48:46.986638625 +0000 UTC m=+1894.982813416" lastFinishedPulling="2026-05-25 11:49:06.710571403 +0000 UTC m=+1914.706746196" observedRunningTime="2026-05-25 11:49:07.07511216 +0000 UTC m=+1915.071286984" watchObservedRunningTime="2026-05-25 11:49:07.076389819 +0000 UTC m=+1915.072564632"
May 25 11:49:08.060027 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:08.059992    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:49:08.060937 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:08.060909    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:49:09.062139 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:09.062100    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:49:14.066125 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:14.066097    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:49:14.066686 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:14.066663    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:49:24.066787 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:24.066726    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:49:34.066849 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:34.066805    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:49:44.066917 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:44.066872    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:49:54.067118 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:49:54.067079    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:50:04.067628 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:04.067540    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:50:14.067531 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:14.067487    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:50:14.565927 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:14.565885    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:50:24.569072 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:24.569045    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:50:32.731594 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.731559    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"]
May 25 11:50:32.732106 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.731956    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" containerID="cri-o://d21c6bd54a4bf5243d21b77da020ef8d3eb0326e3ca69a7ae8149fd8b0906695" gracePeriod=30
May 25 11:50:32.732106 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.732000    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kube-rbac-proxy" containerID="cri-o://142fb5f7a2735aec35179a16ec11ecc678d16a729facfc77f33abe8250faad04" gracePeriod=30
May 25 11:50:32.843315 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.843285    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"]
May 25 11:50:32.846788 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.846743    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:32.849388 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.849368    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
May 25 11:50:32.849478 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.849384    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
May 25 11:50:32.858119 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.858095    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"]
May 25 11:50:32.938975 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.938946    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:32.939150 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.938982    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:32.939150 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.939003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmjkp\" (UniqueName: \"kubernetes.io/projected/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kube-api-access-nmjkp\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:32.939150 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:32.939038    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.039382 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.039354    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.039558 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.039389    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.039558 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.039410    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nmjkp\" (UniqueName: \"kubernetes.io/projected/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kube-api-access-nmjkp\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.039558 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.039432    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.039808 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.039756    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.040126 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.040099    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.041907 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.041884    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.053949 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.053926    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmjkp\" (UniqueName: \"kubernetes.io/projected/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kube-api-access-nmjkp\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-zv766\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.156781 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.156738    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:33.277013 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.276985    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"]
May 25 11:50:33.279449 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:50:33.279424    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bffdce6_9c34_4eb5_9c7a_6b3b00ece360.slice/crio-8275ebb9fc973a7fc2b9b282679e0a67e68ac9387e0f4e682cbffe3b5c0b4f55 WatchSource:0}: Error finding container 8275ebb9fc973a7fc2b9b282679e0a67e68ac9387e0f4e682cbffe3b5c0b4f55: Status 404 returned error can't find the container with id 8275ebb9fc973a7fc2b9b282679e0a67e68ac9387e0f4e682cbffe3b5c0b4f55
May 25 11:50:33.297939 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.297915    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" event={"ID":"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360","Type":"ContainerStarted","Data":"8275ebb9fc973a7fc2b9b282679e0a67e68ac9387e0f4e682cbffe3b5c0b4f55"}
May 25 11:50:33.299607 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.299586    2569 generic.go:358] "Generic (PLEG): container finished" podID="c558365f-3916-415b-b626-00da1e7d645a" containerID="142fb5f7a2735aec35179a16ec11ecc678d16a729facfc77f33abe8250faad04" exitCode=2
May 25 11:50:33.299689 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:33.299653    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" event={"ID":"c558365f-3916-415b-b626-00da1e7d645a","Type":"ContainerDied","Data":"142fb5f7a2735aec35179a16ec11ecc678d16a729facfc77f33abe8250faad04"}
May 25 11:50:34.062923 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:34.062874    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
May 25 11:50:34.305626 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:34.305591    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" event={"ID":"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360","Type":"ContainerStarted","Data":"6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7"}
May 25 11:50:34.566212 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:34.566170    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 25 11:50:37.316578 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.316550    2569 generic.go:358] "Generic (PLEG): container finished" podID="c558365f-3916-415b-b626-00da1e7d645a" containerID="d21c6bd54a4bf5243d21b77da020ef8d3eb0326e3ca69a7ae8149fd8b0906695" exitCode=0
May 25 11:50:37.316947 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.316613    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" event={"ID":"c558365f-3916-415b-b626-00da1e7d645a","Type":"ContainerDied","Data":"d21c6bd54a4bf5243d21b77da020ef8d3eb0326e3ca69a7ae8149fd8b0906695"}
May 25 11:50:37.317967 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.317943    2569 generic.go:358] "Generic (PLEG): container finished" podID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerID="6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7" exitCode=0
May 25 11:50:37.318063 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.317979    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" event={"ID":"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360","Type":"ContainerDied","Data":"6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7"}
May 25 11:50:37.378216 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.378196    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:50:37.473483 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.473451    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c558365f-3916-415b-b626-00da1e7d645a-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"c558365f-3916-415b-b626-00da1e7d645a\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") "
May 25 11:50:37.473582 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.473545    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558365f-3916-415b-b626-00da1e7d645a-kserve-provision-location\") pod \"c558365f-3916-415b-b626-00da1e7d645a\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") "
May 25 11:50:37.473582 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.473569    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c558365f-3916-415b-b626-00da1e7d645a-proxy-tls\") pod \"c558365f-3916-415b-b626-00da1e7d645a\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") "
May 25 11:50:37.473670 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.473594    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6pw9\" (UniqueName: \"kubernetes.io/projected/c558365f-3916-415b-b626-00da1e7d645a-kube-api-access-w6pw9\") pod \"c558365f-3916-415b-b626-00da1e7d645a\" (UID: \"c558365f-3916-415b-b626-00da1e7d645a\") "
May 25 11:50:37.473859 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.473829    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c558365f-3916-415b-b626-00da1e7d645a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c558365f-3916-415b-b626-00da1e7d645a" (UID: "c558365f-3916-415b-b626-00da1e7d645a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:50:37.473859 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.473842    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c558365f-3916-415b-b626-00da1e7d645a-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "c558365f-3916-415b-b626-00da1e7d645a" (UID: "c558365f-3916-415b-b626-00da1e7d645a"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:50:37.475376 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.475352    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c558365f-3916-415b-b626-00da1e7d645a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c558365f-3916-415b-b626-00da1e7d645a" (UID: "c558365f-3916-415b-b626-00da1e7d645a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:50:37.475489 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.475465    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c558365f-3916-415b-b626-00da1e7d645a-kube-api-access-w6pw9" (OuterVolumeSpecName: "kube-api-access-w6pw9") pod "c558365f-3916-415b-b626-00da1e7d645a" (UID: "c558365f-3916-415b-b626-00da1e7d645a"). InnerVolumeSpecName "kube-api-access-w6pw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:50:37.574347 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.574311    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c558365f-3916-415b-b626-00da1e7d645a-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:50:37.574347 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.574342    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558365f-3916-415b-b626-00da1e7d645a-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:50:37.574347 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.574352    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c558365f-3916-415b-b626-00da1e7d645a-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:50:37.574569 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:37.574363    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6pw9\" (UniqueName: \"kubernetes.io/projected/c558365f-3916-415b-b626-00da1e7d645a-kube-api-access-w6pw9\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:50:38.323257 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.323223    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"
May 25 11:50:38.323742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.323223    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz" event={"ID":"c558365f-3916-415b-b626-00da1e7d645a","Type":"ContainerDied","Data":"59c035bb56060f0e77c810e657c9e0f01481a18c04f8c8ca5f5e7f5d10e2cf26"}
May 25 11:50:38.323742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.323358    2569 scope.go:117] "RemoveContainer" containerID="142fb5f7a2735aec35179a16ec11ecc678d16a729facfc77f33abe8250faad04"
May 25 11:50:38.325080 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.325058    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" event={"ID":"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360","Type":"ContainerStarted","Data":"2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854"}
May 25 11:50:38.325179 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.325087    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" event={"ID":"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360","Type":"ContainerStarted","Data":"47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f"}
May 25 11:50:38.325329 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.325306    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:38.331208 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.331190    2569 scope.go:117] "RemoveContainer" containerID="d21c6bd54a4bf5243d21b77da020ef8d3eb0326e3ca69a7ae8149fd8b0906695"
May 25 11:50:38.337852 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.337837    2569 scope.go:117] "RemoveContainer" containerID="ff19c21eaab49879715c58ea751b7297ae5328b744533ef2328a92f44906b56e"
May 25 11:50:38.346277 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.346236    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podStartSLOduration=6.346221716 podStartE2EDuration="6.346221716s" podCreationTimestamp="2026-05-25 11:50:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:50:38.345394039 +0000 UTC m=+2006.341568864" watchObservedRunningTime="2026-05-25 11:50:38.346221716 +0000 UTC m=+2006.342396531"
May 25 11:50:38.357611 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.357586    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"]
May 25 11:50:38.360180 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.360159    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-zgkgz"]
May 25 11:50:38.569107 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:38.569078    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c558365f-3916-415b-b626-00da1e7d645a" path="/var/lib/kubelet/pods/c558365f-3916-415b-b626-00da1e7d645a/volumes"
May 25 11:50:39.329587 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:39.329560    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:39.330785 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:39.330746    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:50:40.332950 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:40.332909    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:50:45.336820 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:45.336791    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:50:45.337368 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:45.337339    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:50:55.337320 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:50:55.337282    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:51:05.337784 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:51:05.337728    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:51:15.337884 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:51:15.337842    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:51:25.337455 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:51:25.337367    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:51:35.337529 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:51:35.337487    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:51:45.337915 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:51:45.337876    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:51:55.337979 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:51:55.337944    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:52:02.951702 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:02.951665    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"]
May 25 11:52:02.952090 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:02.951974    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" containerID="cri-o://47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f" gracePeriod=30
May 25 11:52:02.952090 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:02.952022    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kube-rbac-proxy" containerID="cri-o://2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854" gracePeriod=30
May 25 11:52:03.057818 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.057760    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"]
May 25 11:52:03.058066 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.058054    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kube-rbac-proxy"
May 25 11:52:03.058108 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.058068    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kube-rbac-proxy"
May 25 11:52:03.058108 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.058078    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container"
May 25 11:52:03.058108 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.058083    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container"
May 25 11:52:03.058108 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.058093    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="storage-initializer"
May 25 11:52:03.058108 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.058099    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="storage-initializer"
May 25 11:52:03.058256 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.058144    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kserve-container"
May 25 11:52:03.058256 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.058153    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c558365f-3916-415b-b626-00da1e7d645a" containerName="kube-rbac-proxy"
May 25 11:52:03.061190 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.061170    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.063576 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.063553    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
May 25 11:52:03.063684 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.063584    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
May 25 11:52:03.072001 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.071981    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"]
May 25 11:52:03.082719 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.082697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.082826 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.082743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.082826 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.082787    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.082826 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.082809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdr7z\" (UniqueName: \"kubernetes.io/projected/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kube-api-access-qdr7z\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.183538 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.183504    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.183688 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.183544    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.183688 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.183565    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qdr7z\" (UniqueName: \"kubernetes.io/projected/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kube-api-access-qdr7z\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.183688 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.183601    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.183978 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.183957    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.184195 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.184169    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.185942 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.185921    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.192154 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.192129    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdr7z\" (UniqueName: \"kubernetes.io/projected/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kube-api-access-qdr7z\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.372005 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.371971    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:03.495279 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.495257    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"]
May 25 11:52:03.497411 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:52:03.497389    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ecbefc_9caf_44b4_82ce_abad9c9fdd35.slice/crio-9b9fa326f17206d72e920a192301f2eb0ad7f13ceb755b2ce1c3d49f4af2356e WatchSource:0}: Error finding container 9b9fa326f17206d72e920a192301f2eb0ad7f13ceb755b2ce1c3d49f4af2356e: Status 404 returned error can't find the container with id 9b9fa326f17206d72e920a192301f2eb0ad7f13ceb755b2ce1c3d49f4af2356e
May 25 11:52:03.571895 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.571863    2569 generic.go:358] "Generic (PLEG): container finished" podID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerID="2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854" exitCode=2
May 25 11:52:03.571984 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.571937    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" event={"ID":"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360","Type":"ContainerDied","Data":"2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854"}
May 25 11:52:03.573257 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.573236    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" event={"ID":"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35","Type":"ContainerStarted","Data":"4774327eccba9d67a967fba2cb563de5ea88fbd26c9ab13ce6b5c5054ac15728"}
May 25 11:52:03.573343 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:03.573262    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" event={"ID":"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35","Type":"ContainerStarted","Data":"9b9fa326f17206d72e920a192301f2eb0ad7f13ceb755b2ce1c3d49f4af2356e"}
May 25 11:52:05.333981 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:05.333938    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
May 25 11:52:05.338290 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:05.338265    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 25 11:52:07.387549 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.387526    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:52:07.409685 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.409658    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") "
May 25 11:52:07.409823 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.409708    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kserve-provision-location\") pod \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") "
May 25 11:52:07.409823 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.409759    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-proxy-tls\") pod \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") "
May 25 11:52:07.409943 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.409828    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmjkp\" (UniqueName: \"kubernetes.io/projected/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kube-api-access-nmjkp\") pod \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\" (UID: \"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360\") "
May 25 11:52:07.410066 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.410044    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" (UID: "0bffdce6-9c34-4eb5-9c7a-6b3b00ece360"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:52:07.410138 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.410074    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" (UID: "0bffdce6-9c34-4eb5-9c7a-6b3b00ece360"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:52:07.411921 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.411890    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" (UID: "0bffdce6-9c34-4eb5-9c7a-6b3b00ece360"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:52:07.412020 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.411931    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kube-api-access-nmjkp" (OuterVolumeSpecName: "kube-api-access-nmjkp") pod "0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" (UID: "0bffdce6-9c34-4eb5-9c7a-6b3b00ece360"). InnerVolumeSpecName "kube-api-access-nmjkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:52:07.511112 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.511020    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmjkp\" (UniqueName: \"kubernetes.io/projected/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kube-api-access-nmjkp\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:52:07.511112 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.511058    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:52:07.511112 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.511072    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:52:07.511112 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.511085    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:52:07.585742 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.585706    2569 generic.go:358] "Generic (PLEG): container finished" podID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerID="47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f" exitCode=0
May 25 11:52:07.585908 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.585817    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"
May 25 11:52:07.585964 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.585805    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" event={"ID":"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360","Type":"ContainerDied","Data":"47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f"}
May 25 11:52:07.586022 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.585958    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766" event={"ID":"0bffdce6-9c34-4eb5-9c7a-6b3b00ece360","Type":"ContainerDied","Data":"8275ebb9fc973a7fc2b9b282679e0a67e68ac9387e0f4e682cbffe3b5c0b4f55"}
May 25 11:52:07.586022 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.585989    2569 scope.go:117] "RemoveContainer" containerID="2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854"
May 25 11:52:07.587201 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.587183    2569 generic.go:358] "Generic (PLEG): container finished" podID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerID="4774327eccba9d67a967fba2cb563de5ea88fbd26c9ab13ce6b5c5054ac15728" exitCode=0
May 25 11:52:07.587308 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.587218    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" event={"ID":"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35","Type":"ContainerDied","Data":"4774327eccba9d67a967fba2cb563de5ea88fbd26c9ab13ce6b5c5054ac15728"}
May 25 11:52:07.596326 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.596296    2569 scope.go:117] "RemoveContainer" containerID="47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f"
May 25 11:52:07.604248 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.604230    2569 scope.go:117] "RemoveContainer" containerID="6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7"
May 25 11:52:07.611950 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.611933    2569 scope.go:117] "RemoveContainer" containerID="2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854"
May 25 11:52:07.612232 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:52:07.612199    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854\": container with ID starting with 2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854 not found: ID does not exist" containerID="2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854"
May 25 11:52:07.612290 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.612245    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854"} err="failed to get container status \"2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854\": rpc error: code = NotFound desc = could not find container \"2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854\": container with ID starting with 2050978dde9eb8784fb0e2d3b55792e83b8c440cb317509a345d4b876646c854 not found: ID does not exist"
May 25 11:52:07.612290 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.612268    2569 scope.go:117] "RemoveContainer" containerID="47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f"
May 25 11:52:07.612496 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:52:07.612479    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f\": container with ID starting with 47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f not found: ID does not exist" containerID="47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f"
May 25 11:52:07.612536 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.612502    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f"} err="failed to get container status \"47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f\": rpc error: code = NotFound desc = could not find container \"47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f\": container with ID starting with 47e38315e289814e6dbd34dad3bc8106fe58a6df4fa695ef30abdf58646e758f not found: ID does not exist"
May 25 11:52:07.612536 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.612526    2569 scope.go:117] "RemoveContainer" containerID="6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7"
May 25 11:52:07.612718 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:52:07.612702    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7\": container with ID starting with 6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7 not found: ID does not exist" containerID="6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7"
May 25 11:52:07.612759 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.612722    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7"} err="failed to get container status \"6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7\": rpc error: code = NotFound desc = could not find container \"6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7\": container with ID starting with 6fe0fc12b0f16f7ed5ccf285aefb82868ef5066baa527180a35dd7b53b10f8e7 not found: ID does not exist"
May 25 11:52:07.621464 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.621443    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"]
May 25 11:52:07.625474 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:07.625455    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-zv766"]
May 25 11:52:08.569568 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:08.569539    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" path="/var/lib/kubelet/pods/0bffdce6-9c34-4eb5-9c7a-6b3b00ece360/volumes"
May 25 11:52:08.592356 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:08.592326    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" event={"ID":"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35","Type":"ContainerStarted","Data":"280abd43416f9db0787bfec830acda273efcb64b4ec7cd9cc6d4b10a56d143c0"}
May 25 11:52:08.592489 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:08.592364    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" event={"ID":"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35","Type":"ContainerStarted","Data":"41c58d80ae2d45d3fe045419fe343ad844e097d5c5c19c51aceb0f611eb8a1fb"}
May 25 11:52:08.592580 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:08.592563    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:08.613963 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:08.613919    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podStartSLOduration=5.613904978 podStartE2EDuration="5.613904978s" podCreationTimestamp="2026-05-25 11:52:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:52:08.61281457 +0000 UTC m=+2096.608989384" watchObservedRunningTime="2026-05-25 11:52:08.613904978 +0000 UTC m=+2096.610079792"
May 25 11:52:09.596102 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:09.596072    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:09.597226 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:09.597198    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:52:10.598473 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:10.598436    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:52:13.134439 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:13.134324    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:52:13.159875 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:13.140619    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:52:15.603145 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:15.603120    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:52:15.603573 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:15.603548    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:52:25.603828 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:25.603759    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:52:35.604159 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:35.604119    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:52:45.604182 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:45.604135    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:52:55.603859 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:52:55.603761    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:53:05.603900 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:05.603861    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:53:15.603986 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:15.603948    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:53:17.566262 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:17.566218    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:53:27.567445 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:27.567413    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:53:33.198045 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.198013    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"]
May 25 11:53:33.198422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.198328    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" containerID="cri-o://41c58d80ae2d45d3fe045419fe343ad844e097d5c5c19c51aceb0f611eb8a1fb" gracePeriod=30
May 25 11:53:33.198422 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.198370    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kube-rbac-proxy" containerID="cri-o://280abd43416f9db0787bfec830acda273efcb64b4ec7cd9cc6d4b10a56d143c0" gracePeriod=30
May 25 11:53:33.314483 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.314458    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"]
May 25 11:53:33.314720 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.314708    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container"
May 25 11:53:33.314762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.314721    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container"
May 25 11:53:33.314762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.314736    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="storage-initializer"
May 25 11:53:33.314762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.314742    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="storage-initializer"
May 25 11:53:33.314762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.314748    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kube-rbac-proxy"
May 25 11:53:33.314762 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.314754    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kube-rbac-proxy"
May 25 11:53:33.314955 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.314816    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kserve-container"
May 25 11:53:33.314955 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.314825    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="0bffdce6-9c34-4eb5-9c7a-6b3b00ece360" containerName="kube-rbac-proxy"
May 25 11:53:33.317520 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.317505    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.320068 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.320049    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
May 25 11:53:33.320181 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.320081    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
May 25 11:53:33.326247 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.326225    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"]
May 25 11:53:33.335537 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.335516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4b9\" (UniqueName: \"kubernetes.io/projected/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kube-api-access-pl4b9\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.335619 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.335551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.335619 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.335572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.335696 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.335620    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.436105 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.436078    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.436212 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.436111    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.436212 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.436136    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.436212 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.436195    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4b9\" (UniqueName: \"kubernetes.io/projected/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kube-api-access-pl4b9\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.436652 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.436628    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.436883 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.436865    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.438509 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.438492    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.444580 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.444552    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4b9\" (UniqueName: \"kubernetes.io/projected/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kube-api-access-pl4b9\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.627568 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.627532    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:33.747981 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.747949    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"]
May 25 11:53:33.750631 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:53:33.750599    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0f0b18_e955_4e3c_94b0_f2dfab9b4966.slice/crio-d8eef4c93852fa548650f562d6cb1f29a9315e4275a9f7f25a9555035a21d282 WatchSource:0}: Error finding container d8eef4c93852fa548650f562d6cb1f29a9315e4275a9f7f25a9555035a21d282: Status 404 returned error can't find the container with id d8eef4c93852fa548650f562d6cb1f29a9315e4275a9f7f25a9555035a21d282
May 25 11:53:33.833606 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.833577    2569 generic.go:358] "Generic (PLEG): container finished" podID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerID="280abd43416f9db0787bfec830acda273efcb64b4ec7cd9cc6d4b10a56d143c0" exitCode=2
May 25 11:53:33.833733 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.833641    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" event={"ID":"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35","Type":"ContainerDied","Data":"280abd43416f9db0787bfec830acda273efcb64b4ec7cd9cc6d4b10a56d143c0"}
May 25 11:53:33.834997 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.834974    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" event={"ID":"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966","Type":"ContainerStarted","Data":"aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81"}
May 25 11:53:33.835082 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:33.835005    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" event={"ID":"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966","Type":"ContainerStarted","Data":"d8eef4c93852fa548650f562d6cb1f29a9315e4275a9f7f25a9555035a21d282"}
May 25 11:53:35.599562 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:35.599519    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
May 25 11:53:37.566304 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:37.566258    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 25 11:53:37.848897 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:37.848807    2569 generic.go:358] "Generic (PLEG): container finished" podID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerID="aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81" exitCode=0
May 25 11:53:37.848897 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:37.848850    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" event={"ID":"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966","Type":"ContainerDied","Data":"aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81"}
May 25 11:53:38.853348 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:38.853265    2569 generic.go:358] "Generic (PLEG): container finished" podID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerID="41c58d80ae2d45d3fe045419fe343ad844e097d5c5c19c51aceb0f611eb8a1fb" exitCode=0
May 25 11:53:38.853348 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:38.853333    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" event={"ID":"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35","Type":"ContainerDied","Data":"41c58d80ae2d45d3fe045419fe343ad844e097d5c5c19c51aceb0f611eb8a1fb"}
May 25 11:53:38.854977 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:38.854954    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" event={"ID":"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966","Type":"ContainerStarted","Data":"e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943"}
May 25 11:53:38.855072 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:38.854989    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" event={"ID":"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966","Type":"ContainerStarted","Data":"70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7"}
May 25 11:53:38.855201 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:38.855186    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:38.880929 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:38.879242    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" podStartSLOduration=5.879227193 podStartE2EDuration="5.879227193s" podCreationTimestamp="2026-05-25 11:53:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:53:38.875810887 +0000 UTC m=+2186.871985702" watchObservedRunningTime="2026-05-25 11:53:38.879227193 +0000 UTC m=+2186.875402008"
May 25 11:53:39.130907 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.130885    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:53:39.174976 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.174946    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") "
May 25 11:53:39.175104 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.174996    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-proxy-tls\") pod \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") "
May 25 11:53:39.175104 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.175016    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdr7z\" (UniqueName: \"kubernetes.io/projected/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kube-api-access-qdr7z\") pod \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") "
May 25 11:53:39.175598 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.175219    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kserve-provision-location\") pod \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\" (UID: \"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35\") "
May 25 11:53:39.175598 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.175325    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" (UID: "e8ecbefc-9caf-44b4-82ce-abad9c9fdd35"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:53:39.175598 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.175467    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" (UID: "e8ecbefc-9caf-44b4-82ce-abad9c9fdd35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:53:39.175598 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.175564    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:53:39.175598 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.175576    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:53:39.177096 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.177074    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kube-api-access-qdr7z" (OuterVolumeSpecName: "kube-api-access-qdr7z") pod "e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" (UID: "e8ecbefc-9caf-44b4-82ce-abad9c9fdd35"). InnerVolumeSpecName "kube-api-access-qdr7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:53:39.177189 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.177077    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" (UID: "e8ecbefc-9caf-44b4-82ce-abad9c9fdd35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:53:39.276900 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.276876    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:53:39.276900 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.276897    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdr7z\" (UniqueName: \"kubernetes.io/projected/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35-kube-api-access-qdr7z\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:53:39.862134 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.862098    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk" event={"ID":"e8ecbefc-9caf-44b4-82ce-abad9c9fdd35","Type":"ContainerDied","Data":"9b9fa326f17206d72e920a192301f2eb0ad7f13ceb755b2ce1c3d49f4af2356e"}
May 25 11:53:39.862564 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.862156    2569 scope.go:117] "RemoveContainer" containerID="280abd43416f9db0787bfec830acda273efcb64b4ec7cd9cc6d4b10a56d143c0"
May 25 11:53:39.862564 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.862112    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"
May 25 11:53:39.862564 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.862369    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:53:39.869707 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.869687    2569 scope.go:117] "RemoveContainer" containerID="41c58d80ae2d45d3fe045419fe343ad844e097d5c5c19c51aceb0f611eb8a1fb"
May 25 11:53:39.876271 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.876256    2569 scope.go:117] "RemoveContainer" containerID="4774327eccba9d67a967fba2cb563de5ea88fbd26c9ab13ce6b5c5054ac15728"
May 25 11:53:39.884603 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.884580    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"]
May 25 11:53:39.887359 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:39.887341    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-p6btk"]
May 25 11:53:40.569675 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:40.569642    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" path="/var/lib/kubelet/pods/e8ecbefc-9caf-44b4-82ce-abad9c9fdd35/volumes"
May 25 11:53:45.872517 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:53:45.872488    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:54:15.873182 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:54:15.873144    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
May 25 11:54:25.873795 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:54:25.873697    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
May 25 11:54:35.873935 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:54:35.873894    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
May 25 11:54:45.873997 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:54:45.873961    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
May 25 11:54:55.876590 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:54:55.876560    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:55:03.390148 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.390111    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"]
May 25 11:55:03.390614 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.390509    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kserve-container" containerID="cri-o://70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7" gracePeriod=30
May 25 11:55:03.390614 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.390545    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kube-rbac-proxy" containerID="cri-o://e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943" gracePeriod=30
May 25 11:55:03.509716 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.509690    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"]
May 25 11:55:03.509982 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.509969    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container"
May 25 11:55:03.510027 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.509984    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container"
May 25 11:55:03.510027 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.509997    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kube-rbac-proxy"
May 25 11:55:03.510027 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.510003    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kube-rbac-proxy"
May 25 11:55:03.510027 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.510012    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="storage-initializer"
May 25 11:55:03.510027 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.510017    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="storage-initializer"
May 25 11:55:03.510181 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.510064    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kserve-container"
May 25 11:55:03.510181 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.510071    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8ecbefc-9caf-44b4-82ce-abad9c9fdd35" containerName="kube-rbac-proxy"
May 25 11:55:03.512965 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.512949    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.516434 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.516416    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
May 25 11:55:03.516750 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.516734    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
May 25 11:55:03.522714 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.522694    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"]
May 25 11:55:03.569743 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.569718    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.569868 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.569760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm4b6\" (UniqueName: \"kubernetes.io/projected/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kube-api-access-qm4b6\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.569868 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.569822    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.569868 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.569841    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.670664 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.670587    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.670664 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.670636    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qm4b6\" (UniqueName: \"kubernetes.io/projected/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kube-api-access-qm4b6\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.670664 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.670661    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.670924 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.670680    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.671130 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.671109    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.671405 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.671382    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.673129 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.673113    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.684809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.684788    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm4b6\" (UniqueName: \"kubernetes.io/projected/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kube-api-access-qm4b6\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.822896 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.822864    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:03.943835 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.943810    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"]
May 25 11:55:03.946370 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:55:03.946337    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d8fd71_7741_4d37_8d18_f8964bc5f07b.slice/crio-bca7f9feea51659a327dd76e49aaab7821f481da91f01a11837c52255a9c0839 WatchSource:0}: Error finding container bca7f9feea51659a327dd76e49aaab7821f481da91f01a11837c52255a9c0839: Status 404 returned error can't find the container with id bca7f9feea51659a327dd76e49aaab7821f481da91f01a11837c52255a9c0839
May 25 11:55:03.948132 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:03.948112    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:55:04.106796 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:04.106750    2569 generic.go:358] "Generic (PLEG): container finished" podID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerID="e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943" exitCode=2
May 25 11:55:04.106958 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:04.106823    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" event={"ID":"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966","Type":"ContainerDied","Data":"e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943"}
May 25 11:55:04.108132 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:04.108111    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" event={"ID":"b9d8fd71-7741-4d37-8d18-f8964bc5f07b","Type":"ContainerStarted","Data":"cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69"}
May 25 11:55:04.108231 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:04.108136    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" event={"ID":"b9d8fd71-7741-4d37-8d18-f8964bc5f07b","Type":"ContainerStarted","Data":"bca7f9feea51659a327dd76e49aaab7821f481da91f01a11837c52255a9c0839"}
May 25 11:55:05.868045 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:05.868000    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
May 25 11:55:05.873199 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:05.873170    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
May 25 11:55:07.721124 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.721101    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:55:07.805285 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.805212    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-proxy-tls\") pod \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") "
May 25 11:55:07.805285 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.805246    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kserve-provision-location\") pod \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") "
May 25 11:55:07.805285 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.805283    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") "
May 25 11:55:07.805489 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.805321    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl4b9\" (UniqueName: \"kubernetes.io/projected/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kube-api-access-pl4b9\") pod \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\" (UID: \"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966\") "
May 25 11:55:07.805585 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.805567    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" (UID: "6a0f0b18-e955-4e3c-94b0-f2dfab9b4966"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:55:07.805653 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.805631    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" (UID: "6a0f0b18-e955-4e3c-94b0-f2dfab9b4966"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:55:07.807227 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.807206    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" (UID: "6a0f0b18-e955-4e3c-94b0-f2dfab9b4966"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:55:07.807393 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.807369    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kube-api-access-pl4b9" (OuterVolumeSpecName: "kube-api-access-pl4b9") pod "6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" (UID: "6a0f0b18-e955-4e3c-94b0-f2dfab9b4966"). InnerVolumeSpecName "kube-api-access-pl4b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:55:07.906228 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.906187    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:55:07.906228 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.906228    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:55:07.906454 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.906243    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:55:07.906454 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:07.906260    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl4b9\" (UniqueName: \"kubernetes.io/projected/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966-kube-api-access-pl4b9\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:55:07.964763 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:55:07.964732    2569 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d8fd71_7741_4d37_8d18_f8964bc5f07b.slice/crio-cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d8fd71_7741_4d37_8d18_f8964bc5f07b.slice/crio-conmon-cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69.scope\": RecentStats: unable to find data in memory cache]"
May 25 11:55:08.124478 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.124383    2569 generic.go:358] "Generic (PLEG): container finished" podID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerID="cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69" exitCode=0
May 25 11:55:08.124478 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.124462    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" event={"ID":"b9d8fd71-7741-4d37-8d18-f8964bc5f07b","Type":"ContainerDied","Data":"cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69"}
May 25 11:55:08.126116 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.126095    2569 generic.go:358] "Generic (PLEG): container finished" podID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerID="70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7" exitCode=0
May 25 11:55:08.126218 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.126156    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" event={"ID":"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966","Type":"ContainerDied","Data":"70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7"}
May 25 11:55:08.126218 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.126171    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"
May 25 11:55:08.126218 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.126187    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh" event={"ID":"6a0f0b18-e955-4e3c-94b0-f2dfab9b4966","Type":"ContainerDied","Data":"d8eef4c93852fa548650f562d6cb1f29a9315e4275a9f7f25a9555035a21d282"}
May 25 11:55:08.126218 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.126203    2569 scope.go:117] "RemoveContainer" containerID="e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943"
May 25 11:55:08.134226 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.134144    2569 scope.go:117] "RemoveContainer" containerID="70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7"
May 25 11:55:08.141207 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.141188    2569 scope.go:117] "RemoveContainer" containerID="aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81"
May 25 11:55:08.148557 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.148541    2569 scope.go:117] "RemoveContainer" containerID="e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943"
May 25 11:55:08.148851 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:55:08.148830    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943\": container with ID starting with e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943 not found: ID does not exist" containerID="e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943"
May 25 11:55:08.148938 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.148857    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943"} err="failed to get container status \"e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943\": rpc error: code = NotFound desc = could not find container \"e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943\": container with ID starting with e07d32f437695dc93d681876c108d8fe9bd26ee96e9ddc7fdf5f3971443c0943 not found: ID does not exist"
May 25 11:55:08.148938 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.148875    2569 scope.go:117] "RemoveContainer" containerID="70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7"
May 25 11:55:08.149155 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:55:08.149139    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7\": container with ID starting with 70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7 not found: ID does not exist" containerID="70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7"
May 25 11:55:08.149287 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.149160    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7"} err="failed to get container status \"70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7\": rpc error: code = NotFound desc = could not find container \"70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7\": container with ID starting with 70b975c4e62607369167fbd36dd86efe65a6241dada7ff3968bcb7ce7d84e9c7 not found: ID does not exist"
May 25 11:55:08.149287 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.149174    2569 scope.go:117] "RemoveContainer" containerID="aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81"
May 25 11:55:08.149425 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:55:08.149406    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81\": container with ID starting with aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81 not found: ID does not exist" containerID="aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81"
May 25 11:55:08.149463 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.149431    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81"} err="failed to get container status \"aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81\": rpc error: code = NotFound desc = could not find container \"aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81\": container with ID starting with aaa62449e44df41748291270c51dfc715678cac48d660be90d11481b1fcedb81 not found: ID does not exist"
May 25 11:55:08.156954 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.156934    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"]
May 25 11:55:08.162649 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.162627    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-l6rxh"]
May 25 11:55:08.569464 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:08.569437    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" path="/var/lib/kubelet/pods/6a0f0b18-e955-4e3c-94b0-f2dfab9b4966/volumes"
May 25 11:55:09.130807 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:09.130758    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" event={"ID":"b9d8fd71-7741-4d37-8d18-f8964bc5f07b","Type":"ContainerStarted","Data":"08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0"}
May 25 11:55:09.131288 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:09.130813    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" event={"ID":"b9d8fd71-7741-4d37-8d18-f8964bc5f07b","Type":"ContainerStarted","Data":"7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8"}
May 25 11:55:09.131288 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:09.131077    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:09.157571 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:09.157532    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podStartSLOduration=6.157521232 podStartE2EDuration="6.157521232s" podCreationTimestamp="2026-05-25 11:55:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:55:09.155903815 +0000 UTC m=+2277.152078629" watchObservedRunningTime="2026-05-25 11:55:09.157521232 +0000 UTC m=+2277.153696045"
May 25 11:55:10.134716 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:10.134685    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:16.142940 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:16.142903    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:55:46.143420 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:46.143374    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.35:8080: connect: connection refused"
May 25 11:55:56.144112 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:55:56.144023    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.35:8080: connect: connection refused"
May 25 11:56:06.143946 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:06.143901    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.35:8080: connect: connection refused"
May 25 11:56:16.143460 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:16.143422    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.35:8080: connect: connection refused"
May 25 11:56:17.565957 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:17.565912    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.35:8080: connect: connection refused"
May 25 11:56:27.569606 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:27.569572    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:56:33.634905 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.634859    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"]
May 25 11:56:33.635261 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.635201    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container" containerID="cri-o://7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8" gracePeriod=30
May 25 11:56:33.635304 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.635242    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kube-rbac-proxy" containerID="cri-o://08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0" gracePeriod=30
May 25 11:56:33.758223 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.758191    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"]
May 25 11:56:33.758462 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.758450    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kserve-container"
May 25 11:56:33.758511 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.758463    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kserve-container"
May 25 11:56:33.758511 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.758476    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kube-rbac-proxy"
May 25 11:56:33.758511 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.758482    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kube-rbac-proxy"
May 25 11:56:33.758511 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.758496    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="storage-initializer"
May 25 11:56:33.758511 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.758503    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="storage-initializer"
May 25 11:56:33.758657 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.758546    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kserve-container"
May 25 11:56:33.758657 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.758558    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a0f0b18-e955-4e3c-94b0-f2dfab9b4966" containerName="kube-rbac-proxy"
May 25 11:56:33.761257 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.761242    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:33.763606 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.763583    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
May 25 11:56:33.763727 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.763583    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
May 25 11:56:33.769966 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.769944    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"]
May 25 11:56:33.915585 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.915513    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa5a642-e24c-491b-abb5-57a245bbdc4f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:33.915585 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.915547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hdf\" (UniqueName: \"kubernetes.io/projected/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kube-api-access-x9hdf\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:33.915585 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.915577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa5a642-e24c-491b-abb5-57a245bbdc4f-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:33.915808 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:33.915678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.016016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.015985    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa5a642-e24c-491b-abb5-57a245bbdc4f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.016184 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.016020    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hdf\" (UniqueName: \"kubernetes.io/projected/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kube-api-access-x9hdf\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.016184 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.016059    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa5a642-e24c-491b-abb5-57a245bbdc4f-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.016184 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.016101    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.016184 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:56:34.016113    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
May 25 11:56:34.016370 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:56:34.016188    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa5a642-e24c-491b-abb5-57a245bbdc4f-proxy-tls podName:3fa5a642-e24c-491b-abb5-57a245bbdc4f nodeName:}" failed. No retries permitted until 2026-05-25 11:56:34.516166117 +0000 UTC m=+2362.512340911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3fa5a642-e24c-491b-abb5-57a245bbdc4f-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" (UID: "3fa5a642-e24c-491b-abb5-57a245bbdc4f") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
May 25 11:56:34.016501 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.016483    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.016725 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.016707    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa5a642-e24c-491b-abb5-57a245bbdc4f-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.024692 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.024669    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hdf\" (UniqueName: \"kubernetes.io/projected/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kube-api-access-x9hdf\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.368008 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.367978    2569 generic.go:358] "Generic (PLEG): container finished" podID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerID="08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0" exitCode=2
May 25 11:56:34.368177 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.368040    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" event={"ID":"b9d8fd71-7741-4d37-8d18-f8964bc5f07b","Type":"ContainerDied","Data":"08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0"}
May 25 11:56:34.519803 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.519756    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa5a642-e24c-491b-abb5-57a245bbdc4f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.522101 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.522079    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa5a642-e24c-491b-abb5-57a245bbdc4f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.671625 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.671535    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:34.792933 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:34.792897    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"]
May 25 11:56:34.796642 ip-10-0-141-158 kubenswrapper[2569]: W0525 11:56:34.796613    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa5a642_e24c_491b_abb5_57a245bbdc4f.slice/crio-e198e7ac3cfbc70277ecf1ca1bb0e01e69ce234e232c3debd4cdeba22e8b5bb6 WatchSource:0}: Error finding container e198e7ac3cfbc70277ecf1ca1bb0e01e69ce234e232c3debd4cdeba22e8b5bb6: Status 404 returned error can't find the container with id e198e7ac3cfbc70277ecf1ca1bb0e01e69ce234e232c3debd4cdeba22e8b5bb6
May 25 11:56:35.372580 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:35.372545    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" event={"ID":"3fa5a642-e24c-491b-abb5-57a245bbdc4f","Type":"ContainerStarted","Data":"14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726"}
May 25 11:56:35.372580 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:35.372580    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" event={"ID":"3fa5a642-e24c-491b-abb5-57a245bbdc4f","Type":"ContainerStarted","Data":"e198e7ac3cfbc70277ecf1ca1bb0e01e69ce234e232c3debd4cdeba22e8b5bb6"}
May 25 11:56:36.138380 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:36.138336    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
May 25 11:56:37.566744 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:37.566705    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.35:8080: connect: connection refused"
May 25 11:56:37.972460 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:37.972440    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:56:38.145920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.145833    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") "
May 25 11:56:38.145920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.145886    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kserve-provision-location\") pod \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") "
May 25 11:56:38.145920 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.145911    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-proxy-tls\") pod \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") "
May 25 11:56:38.146179 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.145932    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm4b6\" (UniqueName: \"kubernetes.io/projected/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kube-api-access-qm4b6\") pod \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\" (UID: \"b9d8fd71-7741-4d37-8d18-f8964bc5f07b\") "
May 25 11:56:38.146238 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.146169    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "b9d8fd71-7741-4d37-8d18-f8964bc5f07b" (UID: "b9d8fd71-7741-4d37-8d18-f8964bc5f07b"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:56:38.146238 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.146216    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9d8fd71-7741-4d37-8d18-f8964bc5f07b" (UID: "b9d8fd71-7741-4d37-8d18-f8964bc5f07b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:56:38.148037 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.148009    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9d8fd71-7741-4d37-8d18-f8964bc5f07b" (UID: "b9d8fd71-7741-4d37-8d18-f8964bc5f07b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:56:38.148037 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.148011    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kube-api-access-qm4b6" (OuterVolumeSpecName: "kube-api-access-qm4b6") pod "b9d8fd71-7741-4d37-8d18-f8964bc5f07b" (UID: "b9d8fd71-7741-4d37-8d18-f8964bc5f07b"). InnerVolumeSpecName "kube-api-access-qm4b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:56:38.246432 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.246401    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:56:38.246432 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.246429    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:56:38.246602 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.246443    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:56:38.246602 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.246457    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm4b6\" (UniqueName: \"kubernetes.io/projected/b9d8fd71-7741-4d37-8d18-f8964bc5f07b-kube-api-access-qm4b6\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:56:38.382561 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.382524    2569 generic.go:358] "Generic (PLEG): container finished" podID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerID="7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8" exitCode=0
May 25 11:56:38.382696 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.382610    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" event={"ID":"b9d8fd71-7741-4d37-8d18-f8964bc5f07b","Type":"ContainerDied","Data":"7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8"}
May 25 11:56:38.382696 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.382642    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"
May 25 11:56:38.382696 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.382650    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c" event={"ID":"b9d8fd71-7741-4d37-8d18-f8964bc5f07b","Type":"ContainerDied","Data":"bca7f9feea51659a327dd76e49aaab7821f481da91f01a11837c52255a9c0839"}
May 25 11:56:38.382696 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.382667    2569 scope.go:117] "RemoveContainer" containerID="08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0"
May 25 11:56:38.390339 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.390320    2569 scope.go:117] "RemoveContainer" containerID="7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8"
May 25 11:56:38.397326 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.397310    2569 scope.go:117] "RemoveContainer" containerID="cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69"
May 25 11:56:38.403846 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.403830    2569 scope.go:117] "RemoveContainer" containerID="08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0"
May 25 11:56:38.404117 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:56:38.404083    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0\": container with ID starting with 08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0 not found: ID does not exist" containerID="08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0"
May 25 11:56:38.404201 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.404122    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0"} err="failed to get container status \"08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0\": rpc error: code = NotFound desc = could not find container \"08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0\": container with ID starting with 08611b8110652665ec81ac0d8bf2ae77b0f25c3e09ffbc8c34e6c6ee07181bd0 not found: ID does not exist"
May 25 11:56:38.404201 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.404137    2569 scope.go:117] "RemoveContainer" containerID="7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8"
May 25 11:56:38.404383 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:56:38.404367    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8\": container with ID starting with 7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8 not found: ID does not exist" containerID="7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8"
May 25 11:56:38.404425 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.404386    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8"} err="failed to get container status \"7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8\": rpc error: code = NotFound desc = could not find container \"7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8\": container with ID starting with 7453292ee81d5ce27d2d3d8c5c61903423ff508f464d6edd28442fe8a43f04a8 not found: ID does not exist"
May 25 11:56:38.404425 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.404399    2569 scope.go:117] "RemoveContainer" containerID="cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69"
May 25 11:56:38.404641 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:56:38.404616    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69\": container with ID starting with cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69 not found: ID does not exist" containerID="cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69"
May 25 11:56:38.404641 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.404635    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69"} err="failed to get container status \"cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69\": rpc error: code = NotFound desc = could not find container \"cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69\": container with ID starting with cf25fe1693c2ff0d046e52c9b4bfa54957a2214ca82149e29c09b9d38ab6fd69 not found: ID does not exist"
May 25 11:56:38.405203 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.405185    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"]
May 25 11:56:38.409263 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.409243    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-jls5c"]
May 25 11:56:38.569668 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:38.569640    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" path="/var/lib/kubelet/pods/b9d8fd71-7741-4d37-8d18-f8964bc5f07b/volumes"
May 25 11:56:39.387650 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:39.387622    2569 generic.go:358] "Generic (PLEG): container finished" podID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerID="14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726" exitCode=0
May 25 11:56:39.387831 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:39.387664    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" event={"ID":"3fa5a642-e24c-491b-abb5-57a245bbdc4f","Type":"ContainerDied","Data":"14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726"}
May 25 11:56:40.392250 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:40.392215    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" event={"ID":"3fa5a642-e24c-491b-abb5-57a245bbdc4f","Type":"ContainerStarted","Data":"8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40"}
May 25 11:56:40.392250 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:40.392250    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" event={"ID":"3fa5a642-e24c-491b-abb5-57a245bbdc4f","Type":"ContainerStarted","Data":"9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6"}
May 25 11:56:40.392693 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:40.392529    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:40.392693 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:40.392555    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:56:40.414061 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:40.414014    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" podStartSLOduration=7.413999435 podStartE2EDuration="7.413999435s" podCreationTimestamp="2026-05-25 11:56:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:56:40.413501584 +0000 UTC m=+2368.409676399" watchObservedRunningTime="2026-05-25 11:56:40.413999435 +0000 UTC m=+2368.410174249"
May 25 11:56:46.401355 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:56:46.401326    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:57:13.155016 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:57:13.154906    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:57:13.161159 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:57:13.161134    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 11:57:16.402484 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:57:16.402445    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused"
May 25 11:57:26.402705 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:57:26.402622    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused"
May 25 11:57:36.402421 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:57:36.402385    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused"
May 25 11:57:46.401924 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:57:46.401885    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused"
May 25 11:57:56.405105 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:57:56.405068    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:58:03.862433 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:03.862395    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"]
May 25 11:58:03.862927 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:03.862803    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kserve-container" containerID="cri-o://9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6" gracePeriod=30
May 25 11:58:03.862927 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:03.862835    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kube-rbac-proxy" containerID="cri-o://8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40" gracePeriod=30
May 25 11:58:04.633686 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:04.633653    2569 generic.go:358] "Generic (PLEG): container finished" podID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerID="8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40" exitCode=2
May 25 11:58:04.633877 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:04.633725    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" event={"ID":"3fa5a642-e24c-491b-abb5-57a245bbdc4f","Type":"ContainerDied","Data":"8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40"}
May 25 11:58:06.396219 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:06.396171    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
May 25 11:58:06.401858 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:06.401823    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused"
May 25 11:58:08.596855 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.596829    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:58:08.645062 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.645027    2569 generic.go:358] "Generic (PLEG): container finished" podID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerID="9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6" exitCode=0
May 25 11:58:08.645211 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.645095    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" event={"ID":"3fa5a642-e24c-491b-abb5-57a245bbdc4f","Type":"ContainerDied","Data":"9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6"}
May 25 11:58:08.645211 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.645116    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9" event={"ID":"3fa5a642-e24c-491b-abb5-57a245bbdc4f","Type":"ContainerDied","Data":"e198e7ac3cfbc70277ecf1ca1bb0e01e69ce234e232c3debd4cdeba22e8b5bb6"}
May 25 11:58:08.645211 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.645130    2569 scope.go:117] "RemoveContainer" containerID="8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40"
May 25 11:58:08.645211 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.645160    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"
May 25 11:58:08.652371 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.652350    2569 scope.go:117] "RemoveContainer" containerID="9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6"
May 25 11:58:08.659164 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.659146    2569 scope.go:117] "RemoveContainer" containerID="14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726"
May 25 11:58:08.665341 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.665323    2569 scope.go:117] "RemoveContainer" containerID="8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40"
May 25 11:58:08.665582 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:58:08.665563    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40\": container with ID starting with 8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40 not found: ID does not exist" containerID="8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40"
May 25 11:58:08.665645 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.665592    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40"} err="failed to get container status \"8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40\": rpc error: code = NotFound desc = could not find container \"8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40\": container with ID starting with 8d5c42e1dbcad76ac8016d0ec44e183bd4e50fc40b8b6127b6824b7ea6ed7d40 not found: ID does not exist"
May 25 11:58:08.665645 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.665610    2569 scope.go:117] "RemoveContainer" containerID="9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6"
May 25 11:58:08.665857 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:58:08.665837    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6\": container with ID starting with 9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6 not found: ID does not exist" containerID="9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6"
May 25 11:58:08.665906 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.665863    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6"} err="failed to get container status \"9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6\": rpc error: code = NotFound desc = could not find container \"9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6\": container with ID starting with 9ecabf8007c0ecf944aa41cba922ac3fb5d0e83c74cc9e5aca9f2d0d07c006e6 not found: ID does not exist"
May 25 11:58:08.665906 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.665876    2569 scope.go:117] "RemoveContainer" containerID="14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726"
May 25 11:58:08.666080 ip-10-0-141-158 kubenswrapper[2569]: E0525 11:58:08.666063    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726\": container with ID starting with 14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726 not found: ID does not exist" containerID="14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726"
May 25 11:58:08.666127 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.666084    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726"} err="failed to get container status \"14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726\": rpc error: code = NotFound desc = could not find container \"14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726\": container with ID starting with 14d53a016c8b1c14446fc5e01ad0fd46b141adcb2cf5439e25b9cbcf4963d726 not found: ID does not exist"
May 25 11:58:08.784251 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.784228    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa5a642-e24c-491b-abb5-57a245bbdc4f-proxy-tls\") pod \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") "
May 25 11:58:08.784364 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.784263    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa5a642-e24c-491b-abb5-57a245bbdc4f-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") "
May 25 11:58:08.784364 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.784316    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kserve-provision-location\") pod \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") "
May 25 11:58:08.784364 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.784348    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9hdf\" (UniqueName: \"kubernetes.io/projected/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kube-api-access-x9hdf\") pod \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\" (UID: \"3fa5a642-e24c-491b-abb5-57a245bbdc4f\") "
May 25 11:58:08.784692 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.784666    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fa5a642-e24c-491b-abb5-57a245bbdc4f" (UID: "3fa5a642-e24c-491b-abb5-57a245bbdc4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:58:08.784809 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.784684    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa5a642-e24c-491b-abb5-57a245bbdc4f-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "3fa5a642-e24c-491b-abb5-57a245bbdc4f" (UID: "3fa5a642-e24c-491b-abb5-57a245bbdc4f"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:58:08.786178 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.786151    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa5a642-e24c-491b-abb5-57a245bbdc4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fa5a642-e24c-491b-abb5-57a245bbdc4f" (UID: "3fa5a642-e24c-491b-abb5-57a245bbdc4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:58:08.786528 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.786504    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kube-api-access-x9hdf" (OuterVolumeSpecName: "kube-api-access-x9hdf") pod "3fa5a642-e24c-491b-abb5-57a245bbdc4f" (UID: "3fa5a642-e24c-491b-abb5-57a245bbdc4f"). InnerVolumeSpecName "kube-api-access-x9hdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:58:08.884985 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.884959    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:58:08.884985 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.884982    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9hdf\" (UniqueName: \"kubernetes.io/projected/3fa5a642-e24c-491b-abb5-57a245bbdc4f-kube-api-access-x9hdf\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:58:08.885128 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.884992    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa5a642-e24c-491b-abb5-57a245bbdc4f-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:58:08.885128 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.885002    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa5a642-e24c-491b-abb5-57a245bbdc4f-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 11:58:08.967215 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.967182    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"]
May 25 11:58:08.973321 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:08.973296    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-mn4n9"]
May 25 11:58:10.569084 ip-10-0-141-158 kubenswrapper[2569]: I0525 11:58:10.569051    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" path="/var/lib/kubelet/pods/3fa5a642-e24c-491b-abb5-57a245bbdc4f/volumes"
May 25 12:02:13.173469 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:02:13.173356    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 12:02:13.180987 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:02:13.180968    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 12:04:44.222268 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222233    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"]
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222503    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222513    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222526    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kube-rbac-proxy"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222531    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kube-rbac-proxy"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222544    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kserve-container"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222550    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kserve-container"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222557    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kube-rbac-proxy"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222562    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kube-rbac-proxy"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222573    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="storage-initializer"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222579    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="storage-initializer"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222586    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="storage-initializer"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222592    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="storage-initializer"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222631    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kube-rbac-proxy"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222638    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kserve-container"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222646    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b9d8fd71-7741-4d37-8d18-f8964bc5f07b" containerName="kube-rbac-proxy"
May 25 12:04:44.222737 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.222655    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="3fa5a642-e24c-491b-abb5-57a245bbdc4f" containerName="kserve-container"
May 25 12:04:44.225529 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.225511    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.228559 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.228540    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
May 25 12:04:44.228671 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.228635    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
May 25 12:04:44.229840 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.229826    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 12:04:44.229921 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.229884    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:04:44.229977 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.229916    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:04:44.243172 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.243148    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"]
May 25 12:04:44.317405 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.317371    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71837e9-8157-4e83-a379-ebbfb595a673-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.317405 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.317407    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71837e9-8157-4e83-a379-ebbfb595a673-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.317624 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.317446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtkhl\" (UniqueName: \"kubernetes.io/projected/b71837e9-8157-4e83-a379-ebbfb595a673-kube-api-access-rtkhl\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.317624 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.317472    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71837e9-8157-4e83-a379-ebbfb595a673-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.418393 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.418357    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71837e9-8157-4e83-a379-ebbfb595a673-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.418546 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.418407    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71837e9-8157-4e83-a379-ebbfb595a673-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.418546 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.418431    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71837e9-8157-4e83-a379-ebbfb595a673-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.418546 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.418488    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rtkhl\" (UniqueName: \"kubernetes.io/projected/b71837e9-8157-4e83-a379-ebbfb595a673-kube-api-access-rtkhl\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.418909 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.418891    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71837e9-8157-4e83-a379-ebbfb595a673-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.419188 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.419169    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71837e9-8157-4e83-a379-ebbfb595a673-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.420886 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.420871    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71837e9-8157-4e83-a379-ebbfb595a673-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.426655 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.426634    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtkhl\" (UniqueName: \"kubernetes.io/projected/b71837e9-8157-4e83-a379-ebbfb595a673-kube-api-access-rtkhl\") pod \"isvc-tensorflow-predictor-6756f669d7-hh89b\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.535022 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.534988    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:44.663628 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.663598    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"]
May 25 12:04:44.666655 ip-10-0-141-158 kubenswrapper[2569]: W0525 12:04:44.666624    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71837e9_8157_4e83_a379_ebbfb595a673.slice/crio-e901baa1c017600be29dd5136c8b4e42b051999d5dfc5ae5c041758df0d6136d WatchSource:0}: Error finding container e901baa1c017600be29dd5136c8b4e42b051999d5dfc5ae5c041758df0d6136d: Status 404 returned error can't find the container with id e901baa1c017600be29dd5136c8b4e42b051999d5dfc5ae5c041758df0d6136d
May 25 12:04:44.668486 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.668468    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:04:44.708505 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:44.708481    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" event={"ID":"b71837e9-8157-4e83-a379-ebbfb595a673","Type":"ContainerStarted","Data":"e901baa1c017600be29dd5136c8b4e42b051999d5dfc5ae5c041758df0d6136d"}
May 25 12:04:45.711930 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:45.711893    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" event={"ID":"b71837e9-8157-4e83-a379-ebbfb595a673","Type":"ContainerStarted","Data":"355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4"}
May 25 12:04:49.724210 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:49.724174    2569 generic.go:358] "Generic (PLEG): container finished" podID="b71837e9-8157-4e83-a379-ebbfb595a673" containerID="355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4" exitCode=0
May 25 12:04:49.724685 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:49.724234    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" event={"ID":"b71837e9-8157-4e83-a379-ebbfb595a673","Type":"ContainerDied","Data":"355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4"}
May 25 12:04:53.738399 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:53.738368    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" event={"ID":"b71837e9-8157-4e83-a379-ebbfb595a673","Type":"ContainerStarted","Data":"8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3"}
May 25 12:04:53.738759 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:53.738405    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" event={"ID":"b71837e9-8157-4e83-a379-ebbfb595a673","Type":"ContainerStarted","Data":"f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2"}
May 25 12:04:53.738759 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:53.738597    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:53.758720 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:53.758673    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podStartSLOduration=6.189489969 podStartE2EDuration="9.758660196s" podCreationTimestamp="2026-05-25 12:04:44 +0000 UTC" firstStartedPulling="2026-05-25 12:04:49.72536508 +0000 UTC m=+2857.721539883" lastFinishedPulling="2026-05-25 12:04:53.294535316 +0000 UTC m=+2861.290710110" observedRunningTime="2026-05-25 12:04:53.757663817 +0000 UTC m=+2861.753838631" watchObservedRunningTime="2026-05-25 12:04:53.758660196 +0000 UTC m=+2861.754835010"
May 25 12:04:54.741862 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:54.741760    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:04:54.742984 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:54.742955    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 25 12:04:55.744803 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:04:55.744751    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 25 12:05:00.749563 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:00.749533    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:05:00.750195 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:00.750167    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 25 12:05:10.750927 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:10.750896    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:05:24.996045 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:24.996009    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"]
May 25 12:05:24.996623 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:24.996452    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy" containerID="cri-o://8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3" gracePeriod=30
May 25 12:05:24.996806 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:24.996757    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kserve-container" containerID="cri-o://f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2" gracePeriod=30
May 25 12:05:25.071904 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.071866    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"]
May 25 12:05:25.075249 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.075232    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.077973 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.077952    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
May 25 12:05:25.078071 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.077952    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
May 25 12:05:25.092099 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.092066    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"]
May 25 12:05:25.101533 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.101503    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87qc\" (UniqueName: \"kubernetes.io/projected/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kube-api-access-x87qc\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.101652 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.101565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.101652 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.101591    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.101652 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.101615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.202813 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.202752    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x87qc\" (UniqueName: \"kubernetes.io/projected/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kube-api-access-x87qc\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.202982 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.202837    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.202982 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.202870    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.202982 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.202893    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.203138 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:05:25.202999    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
May 25 12:05:25.203138 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:05:25.203068    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-proxy-tls podName:d19fcffa-535c-43c9-a8e6-b6c28cc6d133 nodeName:}" failed. No retries permitted until 2026-05-25 12:05:25.703049514 +0000 UTC m=+2893.699224305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-proxy-tls") pod "isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" (UID: "d19fcffa-535c-43c9-a8e6-b6c28cc6d133") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
May 25 12:05:25.203250 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.203221    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.203490 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.203470    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.211602 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.211580    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87qc\" (UniqueName: \"kubernetes.io/projected/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kube-api-access-x87qc\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.707094 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.707055    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.709400 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.709382    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:25.745544 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.745505    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
May 25 12:05:25.828547 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.828515    2569 generic.go:358] "Generic (PLEG): container finished" podID="b71837e9-8157-4e83-a379-ebbfb595a673" containerID="8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3" exitCode=2
May 25 12:05:25.828704 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.828589    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" event={"ID":"b71837e9-8157-4e83-a379-ebbfb595a673","Type":"ContainerDied","Data":"8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3"}
May 25 12:05:25.985001 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:25.984975    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:26.104929 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:26.104895    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"]
May 25 12:05:26.108115 ip-10-0-141-158 kubenswrapper[2569]: W0525 12:05:26.108085    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19fcffa_535c_43c9_a8e6_b6c28cc6d133.slice/crio-aea54547f272acd1ff4b977b020c942ef0a48933da4a7a285285e2e0b0065040 WatchSource:0}: Error finding container aea54547f272acd1ff4b977b020c942ef0a48933da4a7a285285e2e0b0065040: Status 404 returned error can't find the container with id aea54547f272acd1ff4b977b020c942ef0a48933da4a7a285285e2e0b0065040
May 25 12:05:26.833902 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:26.833869    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" event={"ID":"d19fcffa-535c-43c9-a8e6-b6c28cc6d133","Type":"ContainerStarted","Data":"3c25d206fa6fbcada4110c26af0dabdd48289a14b5db94fdfe1be6c39b4fb80c"}
May 25 12:05:26.833902 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:26.833902    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" event={"ID":"d19fcffa-535c-43c9-a8e6-b6c28cc6d133","Type":"ContainerStarted","Data":"aea54547f272acd1ff4b977b020c942ef0a48933da4a7a285285e2e0b0065040"}
May 25 12:05:30.745327 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:30.745280    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
May 25 12:05:30.847637 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:30.847554    2569 generic.go:358] "Generic (PLEG): container finished" podID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerID="3c25d206fa6fbcada4110c26af0dabdd48289a14b5db94fdfe1be6c39b4fb80c" exitCode=0
May 25 12:05:30.847637 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:30.847626    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" event={"ID":"d19fcffa-535c-43c9-a8e6-b6c28cc6d133","Type":"ContainerDied","Data":"3c25d206fa6fbcada4110c26af0dabdd48289a14b5db94fdfe1be6c39b4fb80c"}
May 25 12:05:31.852587 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:31.852553    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" event={"ID":"d19fcffa-535c-43c9-a8e6-b6c28cc6d133","Type":"ContainerStarted","Data":"b8f21d669ad34b8660effb0f9e8adad2c1a7b01023355da2d25fe937278f02e3"}
May 25 12:05:31.852996 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:31.852595    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" event={"ID":"d19fcffa-535c-43c9-a8e6-b6c28cc6d133","Type":"ContainerStarted","Data":"1b22ec3ef07cc453b43f3000af6892c522bdaad8e7a7ee2d1877c1dc55b7d609"}
May 25 12:05:31.852996 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:31.852801    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:31.876752 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:31.876702    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podStartSLOduration=6.87668845 podStartE2EDuration="6.87668845s" podCreationTimestamp="2026-05-25 12:05:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:05:31.87530849 +0000 UTC m=+2899.871483328" watchObservedRunningTime="2026-05-25 12:05:31.87668845 +0000 UTC m=+2899.872863263"
May 25 12:05:32.855927 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:32.855896    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:32.857121 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:32.857096    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 25 12:05:33.858433 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:33.858394    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 25 12:05:35.745500 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:35.745451    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
May 25 12:05:35.745989 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:35.745598    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:05:38.863311 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:38.863284    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:38.863727 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:38.863703    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 25 12:05:40.745819 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:40.745749    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
May 25 12:05:45.745817 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:45.745754    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
May 25 12:05:48.864563 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:48.864525    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:05:50.745296 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:50.745250    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
May 25 12:05:55.632308 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.632284    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:05:55.731009 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.730976    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71837e9-8157-4e83-a379-ebbfb595a673-kserve-provision-location\") pod \"b71837e9-8157-4e83-a379-ebbfb595a673\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") "
May 25 12:05:55.731217 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.731039    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtkhl\" (UniqueName: \"kubernetes.io/projected/b71837e9-8157-4e83-a379-ebbfb595a673-kube-api-access-rtkhl\") pod \"b71837e9-8157-4e83-a379-ebbfb595a673\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") "
May 25 12:05:55.731217 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.731083    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71837e9-8157-4e83-a379-ebbfb595a673-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"b71837e9-8157-4e83-a379-ebbfb595a673\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") "
May 25 12:05:55.731217 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.731136    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71837e9-8157-4e83-a379-ebbfb595a673-proxy-tls\") pod \"b71837e9-8157-4e83-a379-ebbfb595a673\" (UID: \"b71837e9-8157-4e83-a379-ebbfb595a673\") "
May 25 12:05:55.731418 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.731393    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71837e9-8157-4e83-a379-ebbfb595a673-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "b71837e9-8157-4e83-a379-ebbfb595a673" (UID: "b71837e9-8157-4e83-a379-ebbfb595a673"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:05:55.733184 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.733146    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71837e9-8157-4e83-a379-ebbfb595a673-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b71837e9-8157-4e83-a379-ebbfb595a673" (UID: "b71837e9-8157-4e83-a379-ebbfb595a673"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:05:55.733305 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.733196    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71837e9-8157-4e83-a379-ebbfb595a673-kube-api-access-rtkhl" (OuterVolumeSpecName: "kube-api-access-rtkhl") pod "b71837e9-8157-4e83-a379-ebbfb595a673" (UID: "b71837e9-8157-4e83-a379-ebbfb595a673"). InnerVolumeSpecName "kube-api-access-rtkhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:05:55.742979 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.742946    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71837e9-8157-4e83-a379-ebbfb595a673-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b71837e9-8157-4e83-a379-ebbfb595a673" (UID: "b71837e9-8157-4e83-a379-ebbfb595a673"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:55.831901 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.831819    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtkhl\" (UniqueName: \"kubernetes.io/projected/b71837e9-8157-4e83-a379-ebbfb595a673-kube-api-access-rtkhl\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:05:55.831901 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.831849    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71837e9-8157-4e83-a379-ebbfb595a673-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:05:55.831901 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.831862    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71837e9-8157-4e83-a379-ebbfb595a673-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:05:55.831901 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.831871    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71837e9-8157-4e83-a379-ebbfb595a673-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:05:55.925498 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.925466    2569 generic.go:358] "Generic (PLEG): container finished" podID="b71837e9-8157-4e83-a379-ebbfb595a673" containerID="f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2" exitCode=137
May 25 12:05:55.925661 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.925527    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" event={"ID":"b71837e9-8157-4e83-a379-ebbfb595a673","Type":"ContainerDied","Data":"f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2"}
May 25 12:05:55.925661 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.925538    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"
May 25 12:05:55.925661 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.925559    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b" event={"ID":"b71837e9-8157-4e83-a379-ebbfb595a673","Type":"ContainerDied","Data":"e901baa1c017600be29dd5136c8b4e42b051999d5dfc5ae5c041758df0d6136d"}
May 25 12:05:55.925661 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.925579    2569 scope.go:117] "RemoveContainer" containerID="8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3"
May 25 12:05:55.935810 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.935763    2569 scope.go:117] "RemoveContainer" containerID="f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2"
May 25 12:05:55.943459 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.943439    2569 scope.go:117] "RemoveContainer" containerID="355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4"
May 25 12:05:55.949840 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.949815    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"]
May 25 12:05:55.950995 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.950972    2569 scope.go:117] "RemoveContainer" containerID="8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3"
May 25 12:05:55.951248 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:05:55.951224    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3\": container with ID starting with 8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3 not found: ID does not exist" containerID="8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3"
May 25 12:05:55.951309 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.951258    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3"} err="failed to get container status \"8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3\": rpc error: code = NotFound desc = could not find container \"8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3\": container with ID starting with 8ad9cfdbac5ff00af5718b474d9b3d50576de3204f22ad824549ca0468b4aee3 not found: ID does not exist"
May 25 12:05:55.951309 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.951278    2569 scope.go:117] "RemoveContainer" containerID="f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2"
May 25 12:05:55.951493 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:05:55.951474    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2\": container with ID starting with f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2 not found: ID does not exist" containerID="f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2"
May 25 12:05:55.951560 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.951502    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2"} err="failed to get container status \"f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2\": rpc error: code = NotFound desc = could not find container \"f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2\": container with ID starting with f2450ef7883465792a614fc5863382e55810122963bc3d83686cfada6d1d87e2 not found: ID does not exist"
May 25 12:05:55.951560 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.951525    2569 scope.go:117] "RemoveContainer" containerID="355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4"
May 25 12:05:55.951784 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:05:55.951756    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4\": container with ID starting with 355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4 not found: ID does not exist" containerID="355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4"
May 25 12:05:55.951860 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.951798    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4"} err="failed to get container status \"355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4\": rpc error: code = NotFound desc = could not find container \"355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4\": container with ID starting with 355f3ad37b47d1d9a2ec7db2a42518a5a4b7e43e45d41c1d22151699d3abcff4 not found: ID does not exist"
May 25 12:05:55.954301 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:55.954279    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-hh89b"]
May 25 12:05:56.569726 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:05:56.569691    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" path="/var/lib/kubelet/pods/b71837e9-8157-4e83-a379-ebbfb595a673/volumes"
May 25 12:06:05.984547 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:05.984509    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"]
May 25 12:06:05.984949 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:05.984841    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kserve-container" containerID="cri-o://1b22ec3ef07cc453b43f3000af6892c522bdaad8e7a7ee2d1877c1dc55b7d609" gracePeriod=30
May 25 12:06:05.984949 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:05.984888    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy" containerID="cri-o://b8f21d669ad34b8660effb0f9e8adad2c1a7b01023355da2d25fe937278f02e3" gracePeriod=30
May 25 12:06:06.106595 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.106566    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"]
May 25 12:06:06.106884 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.106871    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kserve-container"
May 25 12:06:06.106936 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.106886    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kserve-container"
May 25 12:06:06.106936 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.106904    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="storage-initializer"
May 25 12:06:06.106936 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.106909    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="storage-initializer"
May 25 12:06:06.106936 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.106919    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy"
May 25 12:06:06.106936 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.106924    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy"
May 25 12:06:06.107089 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.106969    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kserve-container"
May 25 12:06:06.107089 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.106978    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b71837e9-8157-4e83-a379-ebbfb595a673" containerName="kube-rbac-proxy"
May 25 12:06:06.109953 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.109937    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.112263 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.112242    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
May 25 12:06:06.112383 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.112246    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
May 25 12:06:06.121931 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.121906    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"]
May 25 12:06:06.212725 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.212680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0b036b0-8398-49cb-a459-7fb9e44723af-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.212725 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.212724    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz9bl\" (UniqueName: \"kubernetes.io/projected/a0b036b0-8398-49cb-a459-7fb9e44723af-kube-api-access-hz9bl\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.212965 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.212764    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0b036b0-8398-49cb-a459-7fb9e44723af-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.212965 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.212801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0b036b0-8398-49cb-a459-7fb9e44723af-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.313199 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.313117    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0b036b0-8398-49cb-a459-7fb9e44723af-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.313199 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.313153    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0b036b0-8398-49cb-a459-7fb9e44723af-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.313357 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.313197    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0b036b0-8398-49cb-a459-7fb9e44723af-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.313357 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.313218    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hz9bl\" (UniqueName: \"kubernetes.io/projected/a0b036b0-8398-49cb-a459-7fb9e44723af-kube-api-access-hz9bl\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.313357 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:06:06.313333    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
May 25 12:06:06.313507 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:06:06.313397    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b036b0-8398-49cb-a459-7fb9e44723af-proxy-tls podName:a0b036b0-8398-49cb-a459-7fb9e44723af nodeName:}" failed. No retries permitted until 2026-05-25 12:06:06.813376098 +0000 UTC m=+2934.809550894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a0b036b0-8398-49cb-a459-7fb9e44723af-proxy-tls") pod "isvc-triton-predictor-84bb65d94b-xwf5b" (UID: "a0b036b0-8398-49cb-a459-7fb9e44723af") : secret "isvc-triton-predictor-serving-cert" not found
May 25 12:06:06.313552 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.313535    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0b036b0-8398-49cb-a459-7fb9e44723af-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.313857 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.313840    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0b036b0-8398-49cb-a459-7fb9e44723af-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.321947 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.321920    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz9bl\" (UniqueName: \"kubernetes.io/projected/a0b036b0-8398-49cb-a459-7fb9e44723af-kube-api-access-hz9bl\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.817736 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.817696    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0b036b0-8398-49cb-a459-7fb9e44723af-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.820072 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.820053    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0b036b0-8398-49cb-a459-7fb9e44723af-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-xwf5b\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:06.960152 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.960113    2569 generic.go:358] "Generic (PLEG): container finished" podID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerID="b8f21d669ad34b8660effb0f9e8adad2c1a7b01023355da2d25fe937278f02e3" exitCode=2
May 25 12:06:06.960316 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:06.960182    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" event={"ID":"d19fcffa-535c-43c9-a8e6-b6c28cc6d133","Type":"ContainerDied","Data":"b8f21d669ad34b8660effb0f9e8adad2c1a7b01023355da2d25fe937278f02e3"}
May 25 12:06:07.019397 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:07.019354    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:06:07.146829 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:07.146796    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"]
May 25 12:06:07.149739 ip-10-0-141-158 kubenswrapper[2569]: W0525 12:06:07.149716    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b036b0_8398_49cb_a459_7fb9e44723af.slice/crio-a56021ae9dadb116d2bb0c5f13115110320c83604c04cca15a80844c2001ddb2 WatchSource:0}: Error finding container a56021ae9dadb116d2bb0c5f13115110320c83604c04cca15a80844c2001ddb2: Status 404 returned error can't find the container with id a56021ae9dadb116d2bb0c5f13115110320c83604c04cca15a80844c2001ddb2
May 25 12:06:07.964395 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:07.964358    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" event={"ID":"a0b036b0-8398-49cb-a459-7fb9e44723af","Type":"ContainerStarted","Data":"9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca"}
May 25 12:06:07.964395 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:07.964396    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" event={"ID":"a0b036b0-8398-49cb-a459-7fb9e44723af","Type":"ContainerStarted","Data":"a56021ae9dadb116d2bb0c5f13115110320c83604c04cca15a80844c2001ddb2"}
May 25 12:06:08.859577 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:08.859536    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
May 25 12:06:11.977186 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:11.977138    2569 generic.go:358] "Generic (PLEG): container finished" podID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerID="9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca" exitCode=0
May 25 12:06:11.977569 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:11.977213    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" event={"ID":"a0b036b0-8398-49cb-a459-7fb9e44723af","Type":"ContainerDied","Data":"9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca"}
May 25 12:06:13.859873 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:13.859299    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
May 25 12:06:18.858863 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:18.858819    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
May 25 12:06:18.859487 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:18.858972    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:06:23.859579 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:23.859086    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
May 25 12:06:28.859638 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:28.859514    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
May 25 12:06:33.859564 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:33.859522    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
May 25 12:06:36.083521 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.083483    2569 generic.go:358] "Generic (PLEG): container finished" podID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerID="1b22ec3ef07cc453b43f3000af6892c522bdaad8e7a7ee2d1877c1dc55b7d609" exitCode=137
May 25 12:06:36.084039 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.083577    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" event={"ID":"d19fcffa-535c-43c9-a8e6-b6c28cc6d133","Type":"ContainerDied","Data":"1b22ec3ef07cc453b43f3000af6892c522bdaad8e7a7ee2d1877c1dc55b7d609"}
May 25 12:06:36.679663 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.679637    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:06:36.784839 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.784679    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-proxy-tls\") pod \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") "
May 25 12:06:36.784839 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.784787    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kserve-provision-location\") pod \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") "
May 25 12:06:36.784839 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.784822    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") "
May 25 12:06:36.785124 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.784858    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x87qc\" (UniqueName: \"kubernetes.io/projected/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kube-api-access-x87qc\") pod \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\" (UID: \"d19fcffa-535c-43c9-a8e6-b6c28cc6d133\") "
May 25 12:06:36.785357 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.785310    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config") pod "d19fcffa-535c-43c9-a8e6-b6c28cc6d133" (UID: "d19fcffa-535c-43c9-a8e6-b6c28cc6d133"). InnerVolumeSpecName "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:06:36.787354 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.787328    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d19fcffa-535c-43c9-a8e6-b6c28cc6d133" (UID: "d19fcffa-535c-43c9-a8e6-b6c28cc6d133"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:06:36.787790 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.787736    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kube-api-access-x87qc" (OuterVolumeSpecName: "kube-api-access-x87qc") pod "d19fcffa-535c-43c9-a8e6-b6c28cc6d133" (UID: "d19fcffa-535c-43c9-a8e6-b6c28cc6d133"). InnerVolumeSpecName "kube-api-access-x87qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:06:36.796060 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.796029    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d19fcffa-535c-43c9-a8e6-b6c28cc6d133" (UID: "d19fcffa-535c-43c9-a8e6-b6c28cc6d133"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:06:36.886060 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.886023    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:06:36.886060 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.886054    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:06:36.886060 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.886064    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87qc\" (UniqueName: \"kubernetes.io/projected/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-kube-api-access-x87qc\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:06:36.886357 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:36.886077    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d19fcffa-535c-43c9-a8e6-b6c28cc6d133-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:06:37.089349 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:37.089111    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5" event={"ID":"d19fcffa-535c-43c9-a8e6-b6c28cc6d133","Type":"ContainerDied","Data":"aea54547f272acd1ff4b977b020c942ef0a48933da4a7a285285e2e0b0065040"}
May 25 12:06:37.089349 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:37.089163    2569 scope.go:117] "RemoveContainer" containerID="b8f21d669ad34b8660effb0f9e8adad2c1a7b01023355da2d25fe937278f02e3"
May 25 12:06:37.089349 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:37.089179    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"
May 25 12:06:37.099742 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:37.099663    2569 scope.go:117] "RemoveContainer" containerID="1b22ec3ef07cc453b43f3000af6892c522bdaad8e7a7ee2d1877c1dc55b7d609"
May 25 12:06:37.110573 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:37.110550    2569 scope.go:117] "RemoveContainer" containerID="3c25d206fa6fbcada4110c26af0dabdd48289a14b5db94fdfe1be6c39b4fb80c"
May 25 12:06:37.121873 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:37.121848    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"]
May 25 12:06:37.128806 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:37.127016    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-ck8v5"]
May 25 12:06:38.571889 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:06:38.571680    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" path="/var/lib/kubelet/pods/d19fcffa-535c-43c9-a8e6-b6c28cc6d133/volumes"
May 25 12:08:05.916489 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:05.916414    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 12:08:05.916489 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:05.916421    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 12:08:07.386711 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:07.386672    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" event={"ID":"a0b036b0-8398-49cb-a459-7fb9e44723af","Type":"ContainerStarted","Data":"74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c"}
May 25 12:08:07.387093 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:07.386716    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" event={"ID":"a0b036b0-8398-49cb-a459-7fb9e44723af","Type":"ContainerStarted","Data":"8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343"}
May 25 12:08:07.387093 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:07.386811    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:08:07.418986 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:07.418941    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" podStartSLOduration=6.7468561430000005 podStartE2EDuration="2m1.418928747s" podCreationTimestamp="2026-05-25 12:06:06 +0000 UTC" firstStartedPulling="2026-05-25 12:06:11.978399926 +0000 UTC m=+2939.974574716" lastFinishedPulling="2026-05-25 12:08:06.650472526 +0000 UTC m=+3054.646647320" observedRunningTime="2026-05-25 12:08:07.417946465 +0000 UTC m=+3055.414121277" watchObservedRunningTime="2026-05-25 12:08:07.418928747 +0000 UTC m=+3055.415103559"
May 25 12:08:08.390039 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:08.390009    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:08:08.391175 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:08.391145    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 25 12:08:09.393204 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:09.393160    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 25 12:08:14.397282 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:14.397249    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:08:14.397908 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:14.397890    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:08:17.762605 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.762566    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"]
May 25 12:08:17.763084 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.762984    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kserve-container" containerID="cri-o://8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343" gracePeriod=30
May 25 12:08:17.763084 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.763008    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kube-rbac-proxy" containerID="cri-o://74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c" gracePeriod=30
May 25 12:08:17.898298 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.898274    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"]
May 25 12:08:17.898549 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.898537    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kserve-container"
May 25 12:08:17.898594 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.898550    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kserve-container"
May 25 12:08:17.898594 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.898568    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy"
May 25 12:08:17.898594 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.898573    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy"
May 25 12:08:17.898594 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.898583    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="storage-initializer"
May 25 12:08:17.898594 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.898588    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="storage-initializer"
May 25 12:08:17.898828 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.898628    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kserve-container"
May 25 12:08:17.898828 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.898637    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="d19fcffa-535c-43c9-a8e6-b6c28cc6d133" containerName="kube-rbac-proxy"
May 25 12:08:17.911688 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.911665    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:17.914729 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.914706    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
May 25 12:08:17.916132 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.916110    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
May 25 12:08:17.916922 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.916899    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"]
May 25 12:08:17.975449 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.975420    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:17.975449 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.975462    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:17.975660 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.975526    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4d6b\" (UniqueName: \"kubernetes.io/projected/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kube-api-access-d4d6b\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:17.975660 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:17.975582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.075974 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.075893    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.075974 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.075933    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.076151 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.076117    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d4d6b\" (UniqueName: \"kubernetes.io/projected/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kube-api-access-d4d6b\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.076191 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.076155    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.076398 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.076378    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.076739 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.076717    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.078250 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.078229    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.084931 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.084908    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4d6b\" (UniqueName: \"kubernetes.io/projected/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kube-api-access-d4d6b\") pod \"isvc-xgboost-predictor-8689c4cfcc-gjqvc\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.223185 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.223152    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:18.346248 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.346174    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"]
May 25 12:08:18.349422 ip-10-0-141-158 kubenswrapper[2569]: W0525 12:08:18.349390    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffde503d_e5eb_4b1c_9c28_1be149cb72f1.slice/crio-28d466a2cc4378d126e375d68ca485a66af0ef81ef07fa4fb066be662d5263fb WatchSource:0}: Error finding container 28d466a2cc4378d126e375d68ca485a66af0ef81ef07fa4fb066be662d5263fb: Status 404 returned error can't find the container with id 28d466a2cc4378d126e375d68ca485a66af0ef81ef07fa4fb066be662d5263fb
May 25 12:08:18.419724 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.419696    2569 generic.go:358] "Generic (PLEG): container finished" podID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerID="74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c" exitCode=2
May 25 12:08:18.419855 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.419784    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" event={"ID":"a0b036b0-8398-49cb-a459-7fb9e44723af","Type":"ContainerDied","Data":"74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c"}
May 25 12:08:18.420750 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:18.420731    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" event={"ID":"ffde503d-e5eb-4b1c-9c28-1be149cb72f1","Type":"ContainerStarted","Data":"28d466a2cc4378d126e375d68ca485a66af0ef81ef07fa4fb066be662d5263fb"}
May 25 12:08:19.394297 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:19.394249    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
May 25 12:08:19.425137 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:19.425101    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" event={"ID":"ffde503d-e5eb-4b1c-9c28-1be149cb72f1","Type":"ContainerStarted","Data":"d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3"}
May 25 12:08:20.417158 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.417134    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:08:20.428719 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.428693    2569 generic.go:358] "Generic (PLEG): container finished" podID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerID="8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343" exitCode=0
May 25 12:08:20.428842 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.428765    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"
May 25 12:08:20.428842 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.428789    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" event={"ID":"a0b036b0-8398-49cb-a459-7fb9e44723af","Type":"ContainerDied","Data":"8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343"}
May 25 12:08:20.428842 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.428832    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b" event={"ID":"a0b036b0-8398-49cb-a459-7fb9e44723af","Type":"ContainerDied","Data":"a56021ae9dadb116d2bb0c5f13115110320c83604c04cca15a80844c2001ddb2"}
May 25 12:08:20.428951 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.428852    2569 scope.go:117] "RemoveContainer" containerID="74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c"
May 25 12:08:20.436225 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.436205    2569 scope.go:117] "RemoveContainer" containerID="8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343"
May 25 12:08:20.445609 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.445591    2569 scope.go:117] "RemoveContainer" containerID="9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca"
May 25 12:08:20.452214 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.452192    2569 scope.go:117] "RemoveContainer" containerID="74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c"
May 25 12:08:20.452450 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:08:20.452432    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c\": container with ID starting with 74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c not found: ID does not exist" containerID="74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c"
May 25 12:08:20.452508 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.452460    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c"} err="failed to get container status \"74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c\": rpc error: code = NotFound desc = could not find container \"74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c\": container with ID starting with 74447f6c17c4357c53aef79fee877ebee9912895536eaa2679e1591e5b79c94c not found: ID does not exist"
May 25 12:08:20.452508 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.452481    2569 scope.go:117] "RemoveContainer" containerID="8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343"
May 25 12:08:20.452719 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:08:20.452698    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343\": container with ID starting with 8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343 not found: ID does not exist" containerID="8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343"
May 25 12:08:20.452816 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.452723    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343"} err="failed to get container status \"8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343\": rpc error: code = NotFound desc = could not find container \"8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343\": container with ID starting with 8a3720dc69084ab4d188288eba3be79ff264a3e9a0cb1eba989d4d5a2cf4c343 not found: ID does not exist"
May 25 12:08:20.452816 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.452747    2569 scope.go:117] "RemoveContainer" containerID="9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca"
May 25 12:08:20.453022 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:08:20.452995    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca\": container with ID starting with 9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca not found: ID does not exist" containerID="9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca"
May 25 12:08:20.453093 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.453019    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca"} err="failed to get container status \"9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca\": rpc error: code = NotFound desc = could not find container \"9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca\": container with ID starting with 9164defa2eb529d366034bdbabbc702be65c3aa5ce03bab5f2e9792820cd0aca not found: ID does not exist"
May 25 12:08:20.495413 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.495334    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz9bl\" (UniqueName: \"kubernetes.io/projected/a0b036b0-8398-49cb-a459-7fb9e44723af-kube-api-access-hz9bl\") pod \"a0b036b0-8398-49cb-a459-7fb9e44723af\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") "
May 25 12:08:20.495413 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.495396    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0b036b0-8398-49cb-a459-7fb9e44723af-proxy-tls\") pod \"a0b036b0-8398-49cb-a459-7fb9e44723af\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") "
May 25 12:08:20.495610 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.495448    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0b036b0-8398-49cb-a459-7fb9e44723af-isvc-triton-kube-rbac-proxy-sar-config\") pod \"a0b036b0-8398-49cb-a459-7fb9e44723af\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") "
May 25 12:08:20.495610 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.495473    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0b036b0-8398-49cb-a459-7fb9e44723af-kserve-provision-location\") pod \"a0b036b0-8398-49cb-a459-7fb9e44723af\" (UID: \"a0b036b0-8398-49cb-a459-7fb9e44723af\") "
May 25 12:08:20.495886 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.495857    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b036b0-8398-49cb-a459-7fb9e44723af-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "a0b036b0-8398-49cb-a459-7fb9e44723af" (UID: "a0b036b0-8398-49cb-a459-7fb9e44723af"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:08:20.496001 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.495919    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b036b0-8398-49cb-a459-7fb9e44723af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0b036b0-8398-49cb-a459-7fb9e44723af" (UID: "a0b036b0-8398-49cb-a459-7fb9e44723af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:08:20.498049 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.498025    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b036b0-8398-49cb-a459-7fb9e44723af-kube-api-access-hz9bl" (OuterVolumeSpecName: "kube-api-access-hz9bl") pod "a0b036b0-8398-49cb-a459-7fb9e44723af" (UID: "a0b036b0-8398-49cb-a459-7fb9e44723af"). InnerVolumeSpecName "kube-api-access-hz9bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:08:20.498130 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.498032    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b036b0-8398-49cb-a459-7fb9e44723af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0b036b0-8398-49cb-a459-7fb9e44723af" (UID: "a0b036b0-8398-49cb-a459-7fb9e44723af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:08:20.596653 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.596622    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz9bl\" (UniqueName: \"kubernetes.io/projected/a0b036b0-8398-49cb-a459-7fb9e44723af-kube-api-access-hz9bl\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:08:20.596653 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.596649    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0b036b0-8398-49cb-a459-7fb9e44723af-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:08:20.596844 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.596662    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0b036b0-8398-49cb-a459-7fb9e44723af-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:08:20.596844 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.596674    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0b036b0-8398-49cb-a459-7fb9e44723af-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:08:20.744984 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.744955    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"]
May 25 12:08:20.748870 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:20.748811    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-xwf5b"]
May 25 12:08:22.436186 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:22.436100    2569 generic.go:358] "Generic (PLEG): container finished" podID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerID="d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3" exitCode=0
May 25 12:08:22.436540 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:22.436178    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" event={"ID":"ffde503d-e5eb-4b1c-9c28-1be149cb72f1","Type":"ContainerDied","Data":"d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3"}
May 25 12:08:22.569672 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:22.569639    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" path="/var/lib/kubelet/pods/a0b036b0-8398-49cb-a459-7fb9e44723af/volumes"
May 25 12:08:41.502202 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:41.502170    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" event={"ID":"ffde503d-e5eb-4b1c-9c28-1be149cb72f1","Type":"ContainerStarted","Data":"a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef"}
May 25 12:08:41.502596 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:41.502210    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" event={"ID":"ffde503d-e5eb-4b1c-9c28-1be149cb72f1","Type":"ContainerStarted","Data":"48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f"}
May 25 12:08:41.502596 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:41.502407    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:41.521988 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:41.521945    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podStartSLOduration=5.81783439 podStartE2EDuration="24.521934911s" podCreationTimestamp="2026-05-25 12:08:17 +0000 UTC" firstStartedPulling="2026-05-25 12:08:22.43745011 +0000 UTC m=+3070.433624900" lastFinishedPulling="2026-05-25 12:08:41.141550627 +0000 UTC m=+3089.137725421" observedRunningTime="2026-05-25 12:08:41.521130912 +0000 UTC m=+3089.517305725" watchObservedRunningTime="2026-05-25 12:08:41.521934911 +0000 UTC m=+3089.518109724"
May 25 12:08:42.504594 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:42.504551    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:42.505636 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:42.505605    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 25 12:08:43.507477 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:43.507434    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 25 12:08:48.511414 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:48.511385    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:08:48.512022 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:48.511964    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 25 12:08:58.511982 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:08:58.511937    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 25 12:09:08.512527 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:08.512485    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 25 12:09:18.512424 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:18.512380    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 25 12:09:28.512000 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:28.511915    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 25 12:09:38.512516 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:38.512473    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 25 12:09:48.512470 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:48.512435    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:09:57.952393 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:57.952357    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"]
May 25 12:09:57.952889 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:57.952810    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" containerID="cri-o://48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f" gracePeriod=30
May 25 12:09:57.952961 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:57.952866    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kube-rbac-proxy" containerID="cri-o://a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef" gracePeriod=30
May 25 12:09:58.508033 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:58.507988    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
May 25 12:09:58.512517 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:58.512496    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 25 12:09:58.709690 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:58.709659    2569 generic.go:358] "Generic (PLEG): container finished" podID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerID="a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef" exitCode=2
May 25 12:09:58.709878 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:09:58.709738    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" event={"ID":"ffde503d-e5eb-4b1c-9c28-1be149cb72f1","Type":"ContainerDied","Data":"a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef"}
May 25 12:10:01.294192 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.294166    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:10:01.436164 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.436060    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kserve-provision-location\") pod \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") "
May 25 12:10:01.436164 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.436111    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4d6b\" (UniqueName: \"kubernetes.io/projected/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kube-api-access-d4d6b\") pod \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") "
May 25 12:10:01.436164 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.436140    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") "
May 25 12:10:01.436164 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.436165    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-proxy-tls\") pod \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\" (UID: \"ffde503d-e5eb-4b1c-9c28-1be149cb72f1\") "
May 25 12:10:01.436508 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.436424    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffde503d-e5eb-4b1c-9c28-1be149cb72f1" (UID: "ffde503d-e5eb-4b1c-9c28-1be149cb72f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:10:01.436609 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.436585    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-isvc-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-kube-rbac-proxy-sar-config") pod "ffde503d-e5eb-4b1c-9c28-1be149cb72f1" (UID: "ffde503d-e5eb-4b1c-9c28-1be149cb72f1"). InnerVolumeSpecName "isvc-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:10:01.438218 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.438196    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffde503d-e5eb-4b1c-9c28-1be149cb72f1" (UID: "ffde503d-e5eb-4b1c-9c28-1be149cb72f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:10:01.438285 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.438245    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kube-api-access-d4d6b" (OuterVolumeSpecName: "kube-api-access-d4d6b") pod "ffde503d-e5eb-4b1c-9c28-1be149cb72f1" (UID: "ffde503d-e5eb-4b1c-9c28-1be149cb72f1"). InnerVolumeSpecName "kube-api-access-d4d6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:10:01.537534 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.537496    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:10:01.537534 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.537530    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4d6b\" (UniqueName: \"kubernetes.io/projected/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-kube-api-access-d4d6b\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:10:01.537534 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.537542    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-isvc-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:10:01.537900 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.537552    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffde503d-e5eb-4b1c-9c28-1be149cb72f1-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:10:01.719986 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.719895    2569 generic.go:358] "Generic (PLEG): container finished" podID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerID="48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f" exitCode=0
May 25 12:10:01.719986 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.719983    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"
May 25 12:10:01.720168 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.719978    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" event={"ID":"ffde503d-e5eb-4b1c-9c28-1be149cb72f1","Type":"ContainerDied","Data":"48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f"}
May 25 12:10:01.720168 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.720094    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc" event={"ID":"ffde503d-e5eb-4b1c-9c28-1be149cb72f1","Type":"ContainerDied","Data":"28d466a2cc4378d126e375d68ca485a66af0ef81ef07fa4fb066be662d5263fb"}
May 25 12:10:01.720168 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.720111    2569 scope.go:117] "RemoveContainer" containerID="a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef"
May 25 12:10:01.728351 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.728333    2569 scope.go:117] "RemoveContainer" containerID="48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f"
May 25 12:10:01.735343 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.735323    2569 scope.go:117] "RemoveContainer" containerID="d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3"
May 25 12:10:01.742169 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.742149    2569 scope.go:117] "RemoveContainer" containerID="a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef"
May 25 12:10:01.742409 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:10:01.742390    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef\": container with ID starting with a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef not found: ID does not exist" containerID="a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef"
May 25 12:10:01.742486 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.742416    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef"} err="failed to get container status \"a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef\": rpc error: code = NotFound desc = could not find container \"a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef\": container with ID starting with a772887fc92397a340c7890457fd5e634d15703ba3e0ab49a7a1b91bcc86bfef not found: ID does not exist"
May 25 12:10:01.742486 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.742438    2569 scope.go:117] "RemoveContainer" containerID="48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f"
May 25 12:10:01.742712 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:10:01.742687    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f\": container with ID starting with 48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f not found: ID does not exist" containerID="48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f"
May 25 12:10:01.742758 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.742720    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f"} err="failed to get container status \"48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f\": rpc error: code = NotFound desc = could not find container \"48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f\": container with ID starting with 48cf5e6b770f820555e014b99d73b43a294e00644e0feb2d63887bdd7443599f not found: ID does not exist"
May 25 12:10:01.742758 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.742743    2569 scope.go:117] "RemoveContainer" containerID="d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3"
May 25 12:10:01.743042 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:10:01.743019    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3\": container with ID starting with d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3 not found: ID does not exist" containerID="d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3"
May 25 12:10:01.743094 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.743051    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3"} err="failed to get container status \"d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3\": rpc error: code = NotFound desc = could not find container \"d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3\": container with ID starting with d00f6dbaf51946f4e5bf580b70f5de5d7701eaea5d6850c05da00612647d7ee3 not found: ID does not exist"
May 25 12:10:01.743280 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.743264    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"]
May 25 12:10:01.747211 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:01.747190    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-gjqvc"]
May 25 12:10:02.569805 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:10:02.569754    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" path="/var/lib/kubelet/pods/ffde503d-e5eb-4b1c-9c28-1be149cb72f1/volumes"
May 25 12:11:38.367234 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367198    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"]
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367457    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kserve-container"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367468    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kserve-container"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367476    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kube-rbac-proxy"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367481    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kube-rbac-proxy"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367490    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="storage-initializer"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367496    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="storage-initializer"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367503    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="storage-initializer"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367509    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="storage-initializer"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367518    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367523    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367532    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kube-rbac-proxy"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367536    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kube-rbac-proxy"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367580    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kube-rbac-proxy"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367587    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kube-rbac-proxy"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367593    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0b036b0-8398-49cb-a459-7fb9e44723af" containerName="kserve-container"
May 25 12:11:38.367766 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.367600    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="ffde503d-e5eb-4b1c-9c28-1be149cb72f1" containerName="kserve-container"
May 25 12:11:38.370585 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.370568    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.373195 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.373160    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
May 25 12:11:38.373296 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.373214    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
May 25 12:11:38.374233 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.374215    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:11:38.374312 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.374222    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 12:11:38.374312 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.374261    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:11:38.382070 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.382051    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"]
May 25 12:11:38.518719 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.518685    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g4dl\" (UniqueName: \"kubernetes.io/projected/730b9c1f-b364-4fc0-8be0-8d078438f06d-kube-api-access-8g4dl\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.518719 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.518721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730b9c1f-b364-4fc0-8be0-8d078438f06d-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.518947 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.518799    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730b9c1f-b364-4fc0-8be0-8d078438f06d-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.518947 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.518821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730b9c1f-b364-4fc0-8be0-8d078438f06d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.619322 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.619236    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730b9c1f-b364-4fc0-8be0-8d078438f06d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.619322 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.619289    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8g4dl\" (UniqueName: \"kubernetes.io/projected/730b9c1f-b364-4fc0-8be0-8d078438f06d-kube-api-access-8g4dl\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.619525 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.619404    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730b9c1f-b364-4fc0-8be0-8d078438f06d-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.619525 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.619478    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730b9c1f-b364-4fc0-8be0-8d078438f06d-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.619798 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.619749    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730b9c1f-b364-4fc0-8be0-8d078438f06d-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.620009 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.619985    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730b9c1f-b364-4fc0-8be0-8d078438f06d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.621896 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.621876    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730b9c1f-b364-4fc0-8be0-8d078438f06d-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.628211 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.628189    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g4dl\" (UniqueName: \"kubernetes.io/projected/730b9c1f-b364-4fc0-8be0-8d078438f06d-kube-api-access-8g4dl\") pod \"isvc-xgboost-runtime-predictor-779db84d9-4698j\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.679764 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.679727    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:38.820466 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.820440    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"]
May 25 12:11:38.824481 ip-10-0-141-158 kubenswrapper[2569]: W0525 12:11:38.824447    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730b9c1f_b364_4fc0_8be0_8d078438f06d.slice/crio-92de11e8925ded49e76946a3d29ba9ab5619e2ef4ad9ffb1b9187fb1d4239842 WatchSource:0}: Error finding container 92de11e8925ded49e76946a3d29ba9ab5619e2ef4ad9ffb1b9187fb1d4239842: Status 404 returned error can't find the container with id 92de11e8925ded49e76946a3d29ba9ab5619e2ef4ad9ffb1b9187fb1d4239842
May 25 12:11:38.826485 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.826465    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:11:38.987831 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.987791    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" event={"ID":"730b9c1f-b364-4fc0-8be0-8d078438f06d","Type":"ContainerStarted","Data":"27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd"}
May 25 12:11:38.988066 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:38.987836    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" event={"ID":"730b9c1f-b364-4fc0-8be0-8d078438f06d","Type":"ContainerStarted","Data":"92de11e8925ded49e76946a3d29ba9ab5619e2ef4ad9ffb1b9187fb1d4239842"}
May 25 12:11:42.999763 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:42.999731    2569 generic.go:358] "Generic (PLEG): container finished" podID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerID="27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd" exitCode=0
May 25 12:11:43.000144 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:42.999805    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" event={"ID":"730b9c1f-b364-4fc0-8be0-8d078438f06d","Type":"ContainerDied","Data":"27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd"}
May 25 12:11:44.004673 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:44.004640    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" event={"ID":"730b9c1f-b364-4fc0-8be0-8d078438f06d","Type":"ContainerStarted","Data":"2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73"}
May 25 12:11:44.004673 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:44.004679    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" event={"ID":"730b9c1f-b364-4fc0-8be0-8d078438f06d","Type":"ContainerStarted","Data":"de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7"}
May 25 12:11:44.005214 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:44.004977    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:44.005214 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:44.005112    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:44.006341 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:44.006316    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 25 12:11:44.023635 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:44.023591    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podStartSLOduration=6.023578503 podStartE2EDuration="6.023578503s" podCreationTimestamp="2026-05-25 12:11:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:11:44.022388463 +0000 UTC m=+3272.018563276" watchObservedRunningTime="2026-05-25 12:11:44.023578503 +0000 UTC m=+3272.019753371"
May 25 12:11:45.008466 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:45.008426    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 25 12:11:50.013032 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:50.013000    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:11:50.013597 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:11:50.013566    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 25 12:12:00.014305 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:00.014267    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 25 12:12:10.014397 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:10.014360    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 25 12:12:20.014310 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:20.014270    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 25 12:12:30.014264 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:30.014175    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 25 12:12:40.013826 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:40.013786    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 25 12:12:50.015038 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:50.015007    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:12:58.453044 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:58.453012    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"]
May 25 12:12:58.453535 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:58.453299    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" containerID="cri-o://de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7" gracePeriod=30
May 25 12:12:58.453535 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:58.453330    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kube-rbac-proxy" containerID="cri-o://2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73" gracePeriod=30
May 25 12:12:59.212746 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:59.212713    2569 generic.go:358] "Generic (PLEG): container finished" podID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerID="2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73" exitCode=2
May 25 12:12:59.212933 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:12:59.212808    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" event={"ID":"730b9c1f-b364-4fc0-8be0-8d078438f06d","Type":"ContainerDied","Data":"2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73"}
May 25 12:13:00.009350 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:00.009309    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
May 25 12:13:00.013765 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:00.013736    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 25 12:13:01.802306 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:01.802281    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:13:01.925890 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:01.925799    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730b9c1f-b364-4fc0-8be0-8d078438f06d-proxy-tls\") pod \"730b9c1f-b364-4fc0-8be0-8d078438f06d\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") "
May 25 12:13:01.925890 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:01.925852    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g4dl\" (UniqueName: \"kubernetes.io/projected/730b9c1f-b364-4fc0-8be0-8d078438f06d-kube-api-access-8g4dl\") pod \"730b9c1f-b364-4fc0-8be0-8d078438f06d\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") "
May 25 12:13:01.925890 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:01.925882    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730b9c1f-b364-4fc0-8be0-8d078438f06d-kserve-provision-location\") pod \"730b9c1f-b364-4fc0-8be0-8d078438f06d\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") "
May 25 12:13:01.926176 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:01.925902    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730b9c1f-b364-4fc0-8be0-8d078438f06d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"730b9c1f-b364-4fc0-8be0-8d078438f06d\" (UID: \"730b9c1f-b364-4fc0-8be0-8d078438f06d\") "
May 25 12:13:01.926233 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:01.926165    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730b9c1f-b364-4fc0-8be0-8d078438f06d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "730b9c1f-b364-4fc0-8be0-8d078438f06d" (UID: "730b9c1f-b364-4fc0-8be0-8d078438f06d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:13:01.926308 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:01.926286    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730b9c1f-b364-4fc0-8be0-8d078438f06d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-runtime-kube-rbac-proxy-sar-config") pod "730b9c1f-b364-4fc0-8be0-8d078438f06d" (UID: "730b9c1f-b364-4fc0-8be0-8d078438f06d"). InnerVolumeSpecName "isvc-xgboost-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:13:01.927968 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:01.927947    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730b9c1f-b364-4fc0-8be0-8d078438f06d-kube-api-access-8g4dl" (OuterVolumeSpecName: "kube-api-access-8g4dl") pod "730b9c1f-b364-4fc0-8be0-8d078438f06d" (UID: "730b9c1f-b364-4fc0-8be0-8d078438f06d"). InnerVolumeSpecName "kube-api-access-8g4dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:13:01.928041 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:01.927969    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730b9c1f-b364-4fc0-8be0-8d078438f06d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "730b9c1f-b364-4fc0-8be0-8d078438f06d" (UID: "730b9c1f-b364-4fc0-8be0-8d078438f06d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:13:02.027312 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.027269    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730b9c1f-b364-4fc0-8be0-8d078438f06d-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:13:02.027312 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.027302    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g4dl\" (UniqueName: \"kubernetes.io/projected/730b9c1f-b364-4fc0-8be0-8d078438f06d-kube-api-access-8g4dl\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:13:02.027312 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.027313    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730b9c1f-b364-4fc0-8be0-8d078438f06d-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:13:02.027558 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.027325    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730b9c1f-b364-4fc0-8be0-8d078438f06d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:13:02.226496 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.226402    2569 generic.go:358] "Generic (PLEG): container finished" podID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerID="de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7" exitCode=0
May 25 12:13:02.226496 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.226473    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" event={"ID":"730b9c1f-b364-4fc0-8be0-8d078438f06d","Type":"ContainerDied","Data":"de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7"}
May 25 12:13:02.226712 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.226500    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j" event={"ID":"730b9c1f-b364-4fc0-8be0-8d078438f06d","Type":"ContainerDied","Data":"92de11e8925ded49e76946a3d29ba9ab5619e2ef4ad9ffb1b9187fb1d4239842"}
May 25 12:13:02.226712 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.226514    2569 scope.go:117] "RemoveContainer" containerID="2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73"
May 25 12:13:02.226712 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.226511    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"
May 25 12:13:02.235675 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.235660    2569 scope.go:117] "RemoveContainer" containerID="de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7"
May 25 12:13:02.242693 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.242671    2569 scope.go:117] "RemoveContainer" containerID="27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd"
May 25 12:13:02.249193 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.249169    2569 scope.go:117] "RemoveContainer" containerID="2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73"
May 25 12:13:02.249477 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:13:02.249456    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73\": container with ID starting with 2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73 not found: ID does not exist" containerID="2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73"
May 25 12:13:02.249576 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.249488    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73"} err="failed to get container status \"2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73\": rpc error: code = NotFound desc = could not find container \"2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73\": container with ID starting with 2e803ece238dc7722fdfb9f16935a9268f1e8173634080a93fa990922fcfae73 not found: ID does not exist"
May 25 12:13:02.249576 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.249513    2569 scope.go:117] "RemoveContainer" containerID="de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7"
May 25 12:13:02.249853 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:13:02.249832    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7\": container with ID starting with de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7 not found: ID does not exist" containerID="de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7"
May 25 12:13:02.249922 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.249858    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7"} err="failed to get container status \"de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7\": rpc error: code = NotFound desc = could not find container \"de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7\": container with ID starting with de1885b682438dcd5e14738f0fd8875eb1748cf915a56d9cafcd211181cc44a7 not found: ID does not exist"
May 25 12:13:02.249922 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.249873    2569 scope.go:117] "RemoveContainer" containerID="27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd"
May 25 12:13:02.250139 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:13:02.250123    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd\": container with ID starting with 27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd not found: ID does not exist" containerID="27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd"
May 25 12:13:02.250187 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.250142    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd"} err="failed to get container status \"27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd\": rpc error: code = NotFound desc = could not find container \"27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd\": container with ID starting with 27e7185cd7e1789bd43161b62355dae37cb1f1509a815c63a1f589f121f614bd not found: ID does not exist"
May 25 12:13:02.250670 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.250650    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"]
May 25 12:13:02.257165 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.257145    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-4698j"]
May 25 12:13:02.570194 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:02.570153    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" path="/var/lib/kubelet/pods/730b9c1f-b364-4fc0-8be0-8d078438f06d/volumes"
May 25 12:13:05.935241 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:05.935212    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 12:13:05.936262 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:05.936239    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 12:13:58.727497 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.727419    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"]
May 25 12:13:58.727967 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.727704    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kube-rbac-proxy"
May 25 12:13:58.727967 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.727718    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kube-rbac-proxy"
May 25 12:13:58.727967 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.727726    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="storage-initializer"
May 25 12:13:58.727967 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.727732    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="storage-initializer"
May 25 12:13:58.727967 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.727748    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container"
May 25 12:13:58.727967 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.727753    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container"
May 25 12:13:58.727967 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.727821    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kserve-container"
May 25 12:13:58.727967 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.727830    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="730b9c1f-b364-4fc0-8be0-8d078438f06d" containerName="kube-rbac-proxy"
May 25 12:13:58.730796 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.730761    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:58.733414 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.733395    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
May 25 12:13:58.733627 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.733613    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:13:58.733680 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.733641    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:13:58.734706 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.734689    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
May 25 12:13:58.734823 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.734723    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 12:13:58.740042 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.739994    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"]
May 25 12:13:58.929411 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.929380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:58.929578 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.929430    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:58.929578 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.929462    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:58.929578 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:58.929488    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kglg\" (UniqueName: \"kubernetes.io/projected/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kube-api-access-7kglg\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.030513 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.030484    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7kglg\" (UniqueName: \"kubernetes.io/projected/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kube-api-access-7kglg\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.030679 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.030526    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.030679 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.030562    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.030679 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.030583    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.031048 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.031027    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.031239 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.031221    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.032930 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.032915    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.039641 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.039613    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kglg\" (UniqueName: \"kubernetes.io/projected/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kube-api-access-7kglg\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.041426 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.041411    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:13:59.163844 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.163810    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"]
May 25 12:13:59.166551 ip-10-0-141-158 kubenswrapper[2569]: W0525 12:13:59.166519    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2313c5e7_9f28_4cfb_9f9f_a7c64f50ecb9.slice/crio-af92ee6287e05705866ad140bda30c92e96cdaba6cbb2adb5a2323ea950d934b WatchSource:0}: Error finding container af92ee6287e05705866ad140bda30c92e96cdaba6cbb2adb5a2323ea950d934b: Status 404 returned error can't find the container with id af92ee6287e05705866ad140bda30c92e96cdaba6cbb2adb5a2323ea950d934b
May 25 12:13:59.380346 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.380260    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" event={"ID":"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9","Type":"ContainerStarted","Data":"064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d"}
May 25 12:13:59.380346 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:13:59.380300    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" event={"ID":"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9","Type":"ContainerStarted","Data":"af92ee6287e05705866ad140bda30c92e96cdaba6cbb2adb5a2323ea950d934b"}
May 25 12:14:03.391916 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:03.391877    2569 generic.go:358] "Generic (PLEG): container finished" podID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerID="064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d" exitCode=0
May 25 12:14:03.392329 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:03.391927    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" event={"ID":"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9","Type":"ContainerDied","Data":"064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d"}
May 25 12:14:04.396899 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:04.396865    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" event={"ID":"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9","Type":"ContainerStarted","Data":"c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee"}
May 25 12:14:04.397264 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:04.396905    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" event={"ID":"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9","Type":"ContainerStarted","Data":"5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323"}
May 25 12:14:04.397264 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:04.397096    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:14:04.416485 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:04.416442    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podStartSLOduration=6.416430561 podStartE2EDuration="6.416430561s" podCreationTimestamp="2026-05-25 12:13:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:14:04.415578456 +0000 UTC m=+3412.411753270" watchObservedRunningTime="2026-05-25 12:14:04.416430561 +0000 UTC m=+3412.412605373"
May 25 12:14:05.399560 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:05.399517    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:14:05.400753 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:05.400726    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 25 12:14:06.402342 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:06.402301    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 25 12:14:11.406955 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:11.406924    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:14:11.407458 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:11.407432    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 25 12:14:21.407854 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:21.407813    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 25 12:14:31.407396 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:31.407350    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 25 12:14:41.407943 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:41.407895    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 25 12:14:51.408361 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:14:51.408323    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 25 12:15:01.408510 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:01.408482    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:15:08.820030 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:08.819993    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"]
May 25 12:15:08.820592 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:08.820388    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container" containerID="cri-o://5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323" gracePeriod=30
May 25 12:15:08.820749 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:08.820672    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kube-rbac-proxy" containerID="cri-o://c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee" gracePeriod=30
May 25 12:15:09.573628 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:09.573595    2569 generic.go:358] "Generic (PLEG): container finished" podID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerID="c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee" exitCode=2
May 25 12:15:09.573819 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:09.573667    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" event={"ID":"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9","Type":"ContainerDied","Data":"c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee"}
May 25 12:15:11.403217 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:11.403180    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
May 25 12:15:11.407385 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:11.407355    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 25 12:15:12.360072 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.360051    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:15:12.468897 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.468811    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-proxy-tls\") pod \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") "
May 25 12:15:12.468897 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.468862    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") "
May 25 12:15:12.468897 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.468894    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kserve-provision-location\") pod \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") "
May 25 12:15:12.469388 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.468962    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kglg\" (UniqueName: \"kubernetes.io/projected/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kube-api-access-7kglg\") pod \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\" (UID: \"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9\") "
May 25 12:15:12.469388 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.469232    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" (UID: "2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:15:12.469388 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.469277    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-isvc-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-kube-rbac-proxy-sar-config") pod "2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" (UID: "2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9"). InnerVolumeSpecName "isvc-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:15:12.470865 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.470838    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kube-api-access-7kglg" (OuterVolumeSpecName: "kube-api-access-7kglg") pod "2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" (UID: "2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9"). InnerVolumeSpecName "kube-api-access-7kglg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:15:12.470934 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.470921    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" (UID: "2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:15:12.571875 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.571840    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-proxy-tls\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:15:12.571875 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.571870    2569 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:15:12.572068 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.571888    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kserve-provision-location\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:15:12.572068 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.571902    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kglg\" (UniqueName: \"kubernetes.io/projected/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9-kube-api-access-7kglg\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:15:12.582812 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.582786    2569 generic.go:358] "Generic (PLEG): container finished" podID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerID="5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323" exitCode=0
May 25 12:15:12.582937 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.582830    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" event={"ID":"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9","Type":"ContainerDied","Data":"5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323"}
May 25 12:15:12.582937 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.582860    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr" event={"ID":"2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9","Type":"ContainerDied","Data":"af92ee6287e05705866ad140bda30c92e96cdaba6cbb2adb5a2323ea950d934b"}
May 25 12:15:12.582937 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.582876    2569 scope.go:117] "RemoveContainer" containerID="c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee"
May 25 12:15:12.582937 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.582890    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"
May 25 12:15:12.590004 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.589987    2569 scope.go:117] "RemoveContainer" containerID="5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323"
May 25 12:15:12.596483 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.596464    2569 scope.go:117] "RemoveContainer" containerID="064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d"
May 25 12:15:12.598978 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.598956    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"]
May 25 12:15:12.603689 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.603668    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-ppkhr"]
May 25 12:15:12.604524 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.604510    2569 scope.go:117] "RemoveContainer" containerID="c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee"
May 25 12:15:12.604763 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:15:12.604748    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee\": container with ID starting with c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee not found: ID does not exist" containerID="c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee"
May 25 12:15:12.604856 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.604787    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee"} err="failed to get container status \"c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee\": rpc error: code = NotFound desc = could not find container \"c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee\": container with ID starting with c15d257b7648640a7b8a047ad49d266c1f0aceef895a144007455b9832ef51ee not found: ID does not exist"
May 25 12:15:12.604856 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.604809    2569 scope.go:117] "RemoveContainer" containerID="5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323"
May 25 12:15:12.605024 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:15:12.605004    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323\": container with ID starting with 5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323 not found: ID does not exist" containerID="5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323"
May 25 12:15:12.605064 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.605031    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323"} err="failed to get container status \"5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323\": rpc error: code = NotFound desc = could not find container \"5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323\": container with ID starting with 5f6bf6c87606a0e2d4748231c0063d15322bd54ed82da0dae570422a6e5cd323 not found: ID does not exist"
May 25 12:15:12.605064 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.605048    2569 scope.go:117] "RemoveContainer" containerID="064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d"
May 25 12:15:12.605267 ip-10-0-141-158 kubenswrapper[2569]: E0525 12:15:12.605253    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d\": container with ID starting with 064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d not found: ID does not exist" containerID="064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d"
May 25 12:15:12.605314 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:12.605270    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d"} err="failed to get container status \"064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d\": rpc error: code = NotFound desc = could not find container \"064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d\": container with ID starting with 064cd4858c624d2e8635645b4943200666cb3610d088be00c04fe758ae3a041d not found: ID does not exist"
May 25 12:15:14.568815 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:15:14.568764    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" path="/var/lib/kubelet/pods/2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9/volumes"
May 25 12:18:05.953753 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:18:05.953722    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 12:18:05.956252 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:18:05.955694    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 12:20:47.181531 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.181493    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xndjg/must-gather-m54sk"]
May 25 12:20:47.182052 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.181741    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kube-rbac-proxy"
May 25 12:20:47.182052 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.181751    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kube-rbac-proxy"
May 25 12:20:47.182052 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.181780    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="storage-initializer"
May 25 12:20:47.182052 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.181786    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="storage-initializer"
May 25 12:20:47.182052 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.181796    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container"
May 25 12:20:47.182052 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.181802    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container"
May 25 12:20:47.182052 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.181846    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kserve-container"
May 25 12:20:47.182052 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.181853    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="2313c5e7-9f28-4cfb-9f9f-a7c64f50ecb9" containerName="kube-rbac-proxy"
May 25 12:20:47.184640 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.184624    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:20:47.187484 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.187461    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xndjg\"/\"kube-root-ca.crt\""
May 25 12:20:47.187630 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.187613    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xndjg\"/\"openshift-service-ca.crt\""
May 25 12:20:47.192298 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.192277    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xndjg/must-gather-m54sk"]
May 25 12:20:47.293592 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.293561    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3df812c8-4a84-4004-9123-f057039e3cb4-must-gather-output\") pod \"must-gather-m54sk\" (UID: \"3df812c8-4a84-4004-9123-f057039e3cb4\") " pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:20:47.293799 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.293600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4s2s\" (UniqueName: \"kubernetes.io/projected/3df812c8-4a84-4004-9123-f057039e3cb4-kube-api-access-h4s2s\") pod \"must-gather-m54sk\" (UID: \"3df812c8-4a84-4004-9123-f057039e3cb4\") " pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:20:47.393915 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.393892    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3df812c8-4a84-4004-9123-f057039e3cb4-must-gather-output\") pod \"must-gather-m54sk\" (UID: \"3df812c8-4a84-4004-9123-f057039e3cb4\") " pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:20:47.394036 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.393937    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h4s2s\" (UniqueName: \"kubernetes.io/projected/3df812c8-4a84-4004-9123-f057039e3cb4-kube-api-access-h4s2s\") pod \"must-gather-m54sk\" (UID: \"3df812c8-4a84-4004-9123-f057039e3cb4\") " pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:20:47.394220 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.394203    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3df812c8-4a84-4004-9123-f057039e3cb4-must-gather-output\") pod \"must-gather-m54sk\" (UID: \"3df812c8-4a84-4004-9123-f057039e3cb4\") " pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:20:47.401414 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.401389    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4s2s\" (UniqueName: \"kubernetes.io/projected/3df812c8-4a84-4004-9123-f057039e3cb4-kube-api-access-h4s2s\") pod \"must-gather-m54sk\" (UID: \"3df812c8-4a84-4004-9123-f057039e3cb4\") " pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:20:47.494246 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.494220    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:20:47.612334 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.612301    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xndjg/must-gather-m54sk"]
May 25 12:20:47.614482 ip-10-0-141-158 kubenswrapper[2569]: W0525 12:20:47.614452    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3df812c8_4a84_4004_9123_f057039e3cb4.slice/crio-e5f6817bf8c4a0b9c94d6f826ad1058ec73aad5d1c6fc6fd86e4bebae66092cc WatchSource:0}: Error finding container e5f6817bf8c4a0b9c94d6f826ad1058ec73aad5d1c6fc6fd86e4bebae66092cc: Status 404 returned error can't find the container with id e5f6817bf8c4a0b9c94d6f826ad1058ec73aad5d1c6fc6fd86e4bebae66092cc
May 25 12:20:47.616059 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:47.616044    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:20:48.463689 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:48.463627    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xndjg/must-gather-m54sk" event={"ID":"3df812c8-4a84-4004-9123-f057039e3cb4","Type":"ContainerStarted","Data":"e5f6817bf8c4a0b9c94d6f826ad1058ec73aad5d1c6fc6fd86e4bebae66092cc"}
May 25 12:20:54.485667 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:54.485623    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xndjg/must-gather-m54sk" event={"ID":"3df812c8-4a84-4004-9123-f057039e3cb4","Type":"ContainerStarted","Data":"401b4848eb2b2a7a9f44103e87b8b4ff21ed320f057e16bb5b17059064fa009a"}
May 25 12:20:54.486166 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:54.485675    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xndjg/must-gather-m54sk" event={"ID":"3df812c8-4a84-4004-9123-f057039e3cb4","Type":"ContainerStarted","Data":"e90ab608334e33d371ef27a6d1bdf3cb7d2e69b7c0f9542988005d4391a57cb3"}
May 25 12:20:54.501545 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:20:54.501493    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-xndjg/must-gather-m54sk" podStartSLOduration=1.4555228439999999 podStartE2EDuration="7.501474705s" podCreationTimestamp="2026-05-25 12:20:47 +0000 UTC" firstStartedPulling="2026-05-25 12:20:47.616181042 +0000 UTC m=+3815.612355837" lastFinishedPulling="2026-05-25 12:20:53.662132905 +0000 UTC m=+3821.658307698" observedRunningTime="2026-05-25 12:20:54.500243593 +0000 UTC m=+3822.496418403" watchObservedRunningTime="2026-05-25 12:20:54.501474705 +0000 UTC m=+3822.497649518"
May 25 12:21:17.556371 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:17.556336    2569 generic.go:358] "Generic (PLEG): container finished" podID="3df812c8-4a84-4004-9123-f057039e3cb4" containerID="e90ab608334e33d371ef27a6d1bdf3cb7d2e69b7c0f9542988005d4391a57cb3" exitCode=0
May 25 12:21:17.556371 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:17.556375    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xndjg/must-gather-m54sk" event={"ID":"3df812c8-4a84-4004-9123-f057039e3cb4","Type":"ContainerDied","Data":"e90ab608334e33d371ef27a6d1bdf3cb7d2e69b7c0f9542988005d4391a57cb3"}
May 25 12:21:17.556865 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:17.556636    2569 scope.go:117] "RemoveContainer" containerID="e90ab608334e33d371ef27a6d1bdf3cb7d2e69b7c0f9542988005d4391a57cb3"
May 25 12:21:17.840512 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:17.840423    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xndjg_must-gather-m54sk_3df812c8-4a84-4004-9123-f057039e3cb4/gather/0.log"
May 25 12:21:21.256699 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:21.256670    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k7sb2_f0bac8d4-5c32-4d69-9b35-2d0200701bea/global-pull-secret-syncer/0.log"
May 25 12:21:21.314739 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:21.314712    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hjknq_b3126fb3-2cba-43c5-9244-4cc915eac03d/konnectivity-agent/0.log"
May 25 12:21:21.462125 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:21.462098    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-158.ec2.internal_c03ce34e78836af46c4b9aafeba94245/haproxy/0.log"
May 25 12:21:23.347692 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.347652    2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-xndjg/must-gather-m54sk"]
May 25 12:21:23.348184 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.347955    2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-xndjg/must-gather-m54sk" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" containerName="copy" containerID="cri-o://401b4848eb2b2a7a9f44103e87b8b4ff21ed320f057e16bb5b17059064fa009a" gracePeriod=2
May 25 12:21:23.350532 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.350499    2569 status_manager.go:895] "Failed to get status for pod" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" pod="openshift-must-gather-xndjg/must-gather-m54sk" err="pods \"must-gather-m54sk\" is forbidden: User \"system:node:ip-10-0-141-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xndjg\": no relationship found between node 'ip-10-0-141-158.ec2.internal' and this object"
May 25 12:21:23.351451 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.351427    2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-xndjg/must-gather-m54sk"]
May 25 12:21:23.575560 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.575538    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xndjg_must-gather-m54sk_3df812c8-4a84-4004-9123-f057039e3cb4/copy/0.log"
May 25 12:21:23.575857 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.575837    2569 generic.go:358] "Generic (PLEG): container finished" podID="3df812c8-4a84-4004-9123-f057039e3cb4" containerID="401b4848eb2b2a7a9f44103e87b8b4ff21ed320f057e16bb5b17059064fa009a" exitCode=143
May 25 12:21:23.575938 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.575921    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5f6817bf8c4a0b9c94d6f826ad1058ec73aad5d1c6fc6fd86e4bebae66092cc"
May 25 12:21:23.577979 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.577965    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xndjg_must-gather-m54sk_3df812c8-4a84-4004-9123-f057039e3cb4/copy/0.log"
May 25 12:21:23.578290 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.578277    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:21:23.580256 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.580236    2569 status_manager.go:895] "Failed to get status for pod" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" pod="openshift-must-gather-xndjg/must-gather-m54sk" err="pods \"must-gather-m54sk\" is forbidden: User \"system:node:ip-10-0-141-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xndjg\": no relationship found between node 'ip-10-0-141-158.ec2.internal' and this object"
May 25 12:21:23.669292 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.669218    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4s2s\" (UniqueName: \"kubernetes.io/projected/3df812c8-4a84-4004-9123-f057039e3cb4-kube-api-access-h4s2s\") pod \"3df812c8-4a84-4004-9123-f057039e3cb4\" (UID: \"3df812c8-4a84-4004-9123-f057039e3cb4\") "
May 25 12:21:23.669292 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.669260    2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3df812c8-4a84-4004-9123-f057039e3cb4-must-gather-output\") pod \"3df812c8-4a84-4004-9123-f057039e3cb4\" (UID: \"3df812c8-4a84-4004-9123-f057039e3cb4\") "
May 25 12:21:23.670785 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.670746    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df812c8-4a84-4004-9123-f057039e3cb4-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "3df812c8-4a84-4004-9123-f057039e3cb4" (UID: "3df812c8-4a84-4004-9123-f057039e3cb4"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:21:23.671443 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.671418    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df812c8-4a84-4004-9123-f057039e3cb4-kube-api-access-h4s2s" (OuterVolumeSpecName: "kube-api-access-h4s2s") pod "3df812c8-4a84-4004-9123-f057039e3cb4" (UID: "3df812c8-4a84-4004-9123-f057039e3cb4"). InnerVolumeSpecName "kube-api-access-h4s2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:21:23.770047 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.770002    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4s2s\" (UniqueName: \"kubernetes.io/projected/3df812c8-4a84-4004-9123-f057039e3cb4-kube-api-access-h4s2s\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:21:23.770047 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:23.770042    2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3df812c8-4a84-4004-9123-f057039e3cb4-must-gather-output\") on node \"ip-10-0-141-158.ec2.internal\" DevicePath \"\""
May 25 12:21:24.569508 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:24.569472    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" path="/var/lib/kubelet/pods/3df812c8-4a84-4004-9123-f057039e3cb4/volumes"
May 25 12:21:24.578322 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:24.578296    2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xndjg/must-gather-m54sk"
May 25 12:21:25.512086 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:25.512053    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwkgz_418654e3-284e-4516-8934-67302c832059/node-exporter/0.log"
May 25 12:21:25.534320 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:25.534293    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwkgz_418654e3-284e-4516-8934-67302c832059/kube-rbac-proxy/0.log"
May 25 12:21:25.555501 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:25.555483    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwkgz_418654e3-284e-4516-8934-67302c832059/init-textfile/0.log"
May 25 12:21:26.041124 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:26.041086    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-9hphj_285638e3-f856-4909-9a8a-37404fc1617f/prometheus-operator/0.log"
May 25 12:21:26.073155 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:26.073121    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-9hphj_285638e3-f856-4909-9a8a-37404fc1617f/kube-rbac-proxy/0.log"
May 25 12:21:28.060466 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.060429    2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"]
May 25 12:21:28.060851 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.060705    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" containerName="gather"
May 25 12:21:28.060851 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.060715    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" containerName="gather"
May 25 12:21:28.060851 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.060724    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" containerName="copy"
May 25 12:21:28.060851 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.060730    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" containerName="copy"
May 25 12:21:28.060851 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.060789    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" containerName="copy"
May 25 12:21:28.060851 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.060797    2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="3df812c8-4a84-4004-9123-f057039e3cb4" containerName="gather"
May 25 12:21:28.065150 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.065127    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.072318 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.072297    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7t57z\"/\"openshift-service-ca.crt\""
May 25 12:21:28.072426 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.072336    2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7t57z\"/\"kube-root-ca.crt\""
May 25 12:21:28.073266 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.073251    2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7t57z\"/\"default-dockercfg-5jlg4\""
May 25 12:21:28.077942 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.077920    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"]
May 25 12:21:28.097231 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.097200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-sys\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.097231 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.097230    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-proc\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.097425 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.097248    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-lib-modules\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.097425 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.097274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-podres\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.097425 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.097350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2gxt\" (UniqueName: \"kubernetes.io/projected/e94d4ea6-f03c-4218-a304-510d0c2e8b42-kube-api-access-r2gxt\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.198385 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.198345    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-podres\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.198572 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.198399    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r2gxt\" (UniqueName: \"kubernetes.io/projected/e94d4ea6-f03c-4218-a304-510d0c2e8b42-kube-api-access-r2gxt\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.198572 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.198426    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-sys\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.198572 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.198441    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-proc\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.198572 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.198460    2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-lib-modules\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.198572 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.198517    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-podres\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.198572 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.198527    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-sys\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.198572 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.198565    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-proc\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.198848 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.198586    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e94d4ea6-f03c-4218-a304-510d0c2e8b42-lib-modules\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.215417 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.215393    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2gxt\" (UniqueName: \"kubernetes.io/projected/e94d4ea6-f03c-4218-a304-510d0c2e8b42-kube-api-access-r2gxt\") pod \"perf-node-gather-daemonset-78vbk\" (UID: \"e94d4ea6-f03c-4218-a304-510d0c2e8b42\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.374894 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.374801    2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.495819 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.495711    2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"]
May 25 12:21:28.498476 ip-10-0-141-158 kubenswrapper[2569]: W0525 12:21:28.498447    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode94d4ea6_f03c_4218_a304_510d0c2e8b42.slice/crio-2fcf05c91fe866b84529bf7366393c0133e5636fd2762fd01a40d3402717cccb WatchSource:0}: Error finding container 2fcf05c91fe866b84529bf7366393c0133e5636fd2762fd01a40d3402717cccb: Status 404 returned error can't find the container with id 2fcf05c91fe866b84529bf7366393c0133e5636fd2762fd01a40d3402717cccb
May 25 12:21:28.589672 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.589633    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk" event={"ID":"e94d4ea6-f03c-4218-a304-510d0c2e8b42","Type":"ContainerStarted","Data":"7a9fb1b172039e55c2c48094c18d77e3f7ebd6c831b76932f27140cebd2c4d74"}
May 25 12:21:28.589672 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.589668    2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk" event={"ID":"e94d4ea6-f03c-4218-a304-510d0c2e8b42","Type":"ContainerStarted","Data":"2fcf05c91fe866b84529bf7366393c0133e5636fd2762fd01a40d3402717cccb"}
May 25 12:21:28.589957 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.589749    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:28.605371 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:28.605320    2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk" podStartSLOduration=0.605307805 podStartE2EDuration="605.307805ms" podCreationTimestamp="2026-05-25 12:21:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:21:28.603924185 +0000 UTC m=+3856.600099000" watchObservedRunningTime="2026-05-25 12:21:28.605307805 +0000 UTC m=+3856.601482618"
May 25 12:21:29.288863 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:29.288832    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gktpf_ebf5f84b-f809-49d2-8c90-bdedd5b862e8/dns/0.log"
May 25 12:21:29.307514 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:29.307479    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gktpf_ebf5f84b-f809-49d2-8c90-bdedd5b862e8/kube-rbac-proxy/0.log"
May 25 12:21:29.349699 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:29.349672    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sg6vv_9fa1e4cb-c587-401c-8dc7-a6bc27de70d3/dns-node-resolver/0.log"
May 25 12:21:29.810557 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:29.810531    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwq6c_b32b422f-cd6e-4bcb-a52a-9c7b3e895676/node-ca/0.log"
May 25 12:21:30.832371 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:30.832330    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vlfq6_f2bbe64f-7412-4907-bd9f-658a5034f66c/serve-healthcheck-canary/0.log"
May 25 12:21:31.236458 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:31.236425    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qqxf_b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc/kube-rbac-proxy/0.log"
May 25 12:21:31.254144 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:31.254116    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qqxf_b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc/exporter/0.log"
May 25 12:21:31.273065 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:31.273044    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qqxf_b9135fe8-f84c-4c3d-869c-2f4c2a5e49bc/extractor/0.log"
May 25 12:21:33.395276 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:33.395239    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b94ff949c-c2xgv_99202b1f-c650-49fb-8bac-201064b5c7ef/manager/0.log"
May 25 12:21:33.415370 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:33.415347    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-plw64_2f859f16-b8d9-4269-8937-a27c56583a24/server/0.log"
May 25 12:21:33.829881 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:33.829849    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nhl42_fcc960dc-a84c-442d-a302-5142a16452db/s3-init/0.log"
May 25 12:21:33.873539 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:33.873495    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4gb85_33fd850e-0a67-445d-87bf-3a3d28918f4a/s3-tls-init-serving/0.log"
May 25 12:21:33.942914 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:33.942888    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-7fd5766db9-vk56t_87be0d53-f438-486f-9178-e7a1b4833432/seaweedfs-tls-serving/0.log"
May 25 12:21:34.602533 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:34.602503    2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-78vbk"
May 25 12:21:38.105864 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:38.105838    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-jj8fs_2f1e16c7-8531-460c-b2d9-013449b660fa/migrator/0.log"
May 25 12:21:38.129888 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:38.129864    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-jj8fs_2f1e16c7-8531-460c-b2d9-013449b660fa/graceful-termination/0.log"
May 25 12:21:39.439237 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:39.439210    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5mdm_9b5d0970-d81c-4ffb-9e1d-d2be86769fb9/kube-multus-additional-cni-plugins/0.log"
May 25 12:21:39.462389 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:39.462363    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5mdm_9b5d0970-d81c-4ffb-9e1d-d2be86769fb9/egress-router-binary-copy/0.log"
May 25 12:21:39.485897 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:39.485875    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5mdm_9b5d0970-d81c-4ffb-9e1d-d2be86769fb9/cni-plugins/0.log"
May 25 12:21:39.506385 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:39.506359    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5mdm_9b5d0970-d81c-4ffb-9e1d-d2be86769fb9/bond-cni-plugin/0.log"
May 25 12:21:39.529929 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:39.529901    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5mdm_9b5d0970-d81c-4ffb-9e1d-d2be86769fb9/routeoverride-cni/0.log"
May 25 12:21:39.549133 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:39.549108    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5mdm_9b5d0970-d81c-4ffb-9e1d-d2be86769fb9/whereabouts-cni-bincopy/0.log"
May 25 12:21:39.568564 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:39.568512    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b5mdm_9b5d0970-d81c-4ffb-9e1d-d2be86769fb9/whereabouts-cni/0.log"
May 25 12:21:39.964719 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:39.964644    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jfr78_295056bc-0e37-4116-bbd8-00f40557f5d5/kube-multus/0.log"
May 25 12:21:40.070108 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:40.070077    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z2l9j_c8d62b8b-fa29-494d-a37a-3436347370ed/network-metrics-daemon/0.log"
May 25 12:21:40.086478 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:40.086447    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z2l9j_c8d62b8b-fa29-494d-a37a-3436347370ed/kube-rbac-proxy/0.log"
May 25 12:21:41.188391 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:41.188359    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-controller/0.log"
May 25 12:21:41.202984 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:41.202955    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/0.log"
May 25 12:21:41.235030 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:41.235000    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovn-acl-logging/1.log"
May 25 12:21:41.257448 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:41.257421    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/kube-rbac-proxy-node/0.log"
May 25 12:21:41.281042 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:41.281020    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/kube-rbac-proxy-ovn-metrics/0.log"
May 25 12:21:41.299995 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:41.299954    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/northd/0.log"
May 25 12:21:41.319114 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:41.319090    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/nbdb/0.log"
May 25 12:21:41.337979 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:41.337962    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/sbdb/0.log"
May 25 12:21:41.521247 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:41.521211    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbtgj_f67dfd38-152e-40bf-beb0-430505aa58fe/ovnkube-controller/0.log"
May 25 12:21:42.868132 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:42.868097    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-gzg8z_607c3221-179d-44be-81d0-930ca62118ad/network-check-target-container/0.log"
May 25 12:21:43.771445 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:43.771413    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5tbtf_28e6f486-3b90-4ab3-8088-d41e46546740/iptables-alerter/0.log"
May 25 12:21:44.468251 ip-10-0-141-158 kubenswrapper[2569]: I0525 12:21:44.468201    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vlsbt_9ab88c23-ea8f-4dc4-8e7a-bc7fe560a605/tuned/0.log"