May 25 12:10:11.076381 ip-10-0-141-91 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 12:10:11.076414 ip-10-0-141-91 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 12:10:11.076423 ip-10-0-141-91 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 12:10:11.076726 ip-10-0-141-91 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 12:10:22.305561 ip-10-0-141-91 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 25 12:10:22.305578 ip-10-0-141-91 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ba4c7ba42dee49648c86eef386c2baee --
May 25 12:12:44.212532 ip-10-0-141-91 systemd[1]: Starting Kubernetes Kubelet...
May 25 12:12:44.700993 ip-10-0-141-91 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:12:44.700993 ip-10-0-141-91 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 12:12:44.702029 ip-10-0-141-91 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:12:44.702029 ip-10-0-141-91 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 12:12:44.702029 ip-10-0-141-91 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:12:44.702029 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.701908    2562 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 12:12:44.709049 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709033    2562 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:44.709049 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709048    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709052    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709055    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709058    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709061    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709064    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709068    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709071    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709074    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709077    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709081    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709085    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709089    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709092    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709095    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709098    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709105    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709108    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709111    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:44.709120 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709113    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709116    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709118    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709121    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709124    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709127    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709130    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709133    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709136    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709138    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709141    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709144    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709148    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709150    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709153    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709155    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709158    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709160    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709163    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709165    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:44.709604 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709168    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709171    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709175    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709177    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709180    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709182    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709185    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709188    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709190    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709192    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709195    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709199    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709203    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709206    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709209    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709212    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709215    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709218    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709220    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709223    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:44.710077 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709226    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709228    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709231    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709233    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709236    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709238    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709241    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709244    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709246    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709249    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709252    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709254    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709270    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709274    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709279    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709284    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709287    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709290    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709292    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709296    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:44.710605 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709299    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709301    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709304    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709306    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709309    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709312    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709695    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709700    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709703    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709706    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709709    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709712    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709715    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709717    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709720    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709723    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709725    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709728    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709731    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709733    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:44.711081 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709736    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709738    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709741    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709743    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709748    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709752    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709755    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709758    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709761    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709765    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709768    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709771    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709773    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709776    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709779    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709782    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709784    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709787    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709789    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:44.711575 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709792    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709794    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709796    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709800    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709802    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709805    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709807    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709810    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709812    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709815    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709817    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709820    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709822    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709824    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709827    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709829    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709832    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709835    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709837    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709840    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:44.712075 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709842    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709845    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709849    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709852    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709855    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709857    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709861    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709864    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709866    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709869    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709872    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709874    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709877    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709880    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709882    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709885    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709887    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709890    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709892    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:44.712603 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709895    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709897    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709900    2562 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709902    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709906    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709909    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709911    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709914    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709916    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709919    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709921    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709924    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709926    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.709929    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710546    2562 flags.go:64] FLAG: --address="0.0.0.0"
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710565    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710572    2562 flags.go:64] FLAG: --anonymous-auth="true"
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710576    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710581    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710585    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710589    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710594    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 12:12:44.713067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710597    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710600    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710604    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710607    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710611    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710614    2562 flags.go:64] FLAG: --cgroup-root=""
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710617    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710620    2562 flags.go:64] FLAG: --client-ca-file=""
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710623    2562 flags.go:64] FLAG: --cloud-config=""
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710626    2562 flags.go:64] FLAG: --cloud-provider="external"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710629    2562 flags.go:64] FLAG: --cluster-dns="[]"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710634    2562 flags.go:64] FLAG: --cluster-domain=""
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710637    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710641    2562 flags.go:64] FLAG: --config-dir=""
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710644    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710647    2562 flags.go:64] FLAG: --container-log-max-files="5"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710652    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710655    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710658    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710661    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710664    2562 flags.go:64] FLAG: --contention-profiling="false"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710667    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710670    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710673    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710676    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 12:12:44.713608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710681    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710685    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710688    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710691    2562 flags.go:64] FLAG: --enable-load-reader="false"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710695    2562 flags.go:64] FLAG: --enable-server="true"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710698    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710703    2562 flags.go:64] FLAG: --event-burst="100"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710706    2562 flags.go:64] FLAG: --event-qps="50"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710709    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710712    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710715    2562 flags.go:64] FLAG: --eviction-hard=""
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710719    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710722    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710725    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710728    2562 flags.go:64] FLAG: --eviction-soft=""
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710731    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710734    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710737    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710740    2562 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710743    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710746    2562 flags.go:64] FLAG: --fail-swap-on="true"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710749    2562 flags.go:64] FLAG: --feature-gates=""
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710754    2562 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710757    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710760    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 12:12:44.714212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710763    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710766    2562 flags.go:64] FLAG: --healthz-port="10248"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710769    2562 flags.go:64] FLAG: --help="false"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710772    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-141-91.ec2.internal"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710775    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710778    2562 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710781    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710785    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710788    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710792    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710795    2562 flags.go:64] FLAG: --image-service-endpoint=""
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710798    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710801    2562 flags.go:64] FLAG: --kube-api-burst="100"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710804    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710807    2562 flags.go:64] FLAG: --kube-api-qps="50"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710811    2562 flags.go:64] FLAG: --kube-reserved=""
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710814    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710816    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710819    2562 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710822    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710825    2562 flags.go:64] FLAG: --lock-file=""
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710828    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710831    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710834    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 12:12:44.714829 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710840    2562 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710843    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710846    2562 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710848    2562 flags.go:64] FLAG: --logging-format="text"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710851    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710855    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710857    2562 flags.go:64] FLAG: --manifest-url=""
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710860    2562 flags.go:64] FLAG: --manifest-url-header=""
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710865    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710868    2562 flags.go:64] FLAG: --max-open-files="1000000"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710872    2562 flags.go:64] FLAG: --max-pods="110"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710875    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710878    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710881    2562 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710884    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710887    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710891    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710894    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710902    2562 flags.go:64] FLAG: --node-status-max-images="50"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710907    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710910    2562 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710914    2562 flags.go:64] FLAG: --pod-cidr=""
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710916    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 12:12:44.715430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710923    2562 flags.go:64] FLAG: --pod-manifest-path=""
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710925    2562 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710928    2562 flags.go:64] FLAG: --pods-per-core="0"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710931    2562 flags.go:64] FLAG: --port="10250"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710934    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710953    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09ab728d90129b519"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710956    2562 flags.go:64] FLAG: --qos-reserved=""
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710960    2562 flags.go:64] FLAG: --read-only-port="10255"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710963    2562 flags.go:64] FLAG: --register-node="true"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710967    2562 flags.go:64] FLAG: --register-schedulable="true"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710970    2562 flags.go:64] FLAG: --register-with-taints=""
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710974    2562 flags.go:64] FLAG: --registry-burst="10"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710977    2562 flags.go:64] FLAG: --registry-qps="5"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710980    2562 flags.go:64] FLAG: --reserved-cpus=""
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710983    2562 flags.go:64] FLAG: --reserved-memory=""
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710987    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710990    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710993    2562 flags.go:64] FLAG: --rotate-certificates="false"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710996    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.710999    2562 flags.go:64] FLAG: --runonce="false"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711002    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711005    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711008    2562 flags.go:64] FLAG: --seccomp-default="false"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711011    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711014    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711017    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 12:12:44.715986 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711020    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711023    2562 flags.go:64] FLAG: --storage-driver-password="root"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711027    2562 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711031    2562 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711034    2562 flags.go:64] FLAG: --storage-driver-user="root"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711037    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711040    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711043    2562 flags.go:64] FLAG: --system-cgroups=""
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711046    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711051    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711054    2562 flags.go:64] FLAG: --tls-cert-file=""
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711057    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711062    2562 flags.go:64] FLAG: --tls-min-version=""
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711065    2562 flags.go:64] FLAG: --tls-private-key-file=""
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711067    2562 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711070    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711073    2562 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711076    2562 flags.go:64] FLAG: --v="2"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711083    2562 flags.go:64] FLAG: --version="false"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711087    2562 flags.go:64] FLAG: --vmodule=""
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711092    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.711095    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711204    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711209    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:44.716619 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711213    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711217    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711220    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711223    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711226    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711229    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711234    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711236    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711239    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711242    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711245    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711249    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711251    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711254    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711269    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711272    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711275    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711278    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711280    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:44.717206 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711283    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711286    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711289    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711291    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711294    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711297    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711299    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711302    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711304    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711307    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711309    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711312    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711315    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711317    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711319    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711322    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711324    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711327    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711329    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711334    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:44.717780 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711337    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711339    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711342    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711345    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711348    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711351    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711353    2562 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711356    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711358    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711361    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711363    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711366    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711368    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711371    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711373    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711376    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711378    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711381    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711383    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711386    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:44.718291 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711389    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711391    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711394    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711396    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711399    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711402    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711404    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711406    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711409    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711413    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711416    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711420    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711423    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711426    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711429    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711432    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711435    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711438    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711441    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:44.718771 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711443    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711446    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711448    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711451    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711454    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.711456    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.712269    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.718853    2562 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.718868    2562 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718917    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718923    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718926    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718930    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718933    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718936    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718939    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:44.719234 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718942    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718945    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718947    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718950    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718953    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718955    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718958    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718960    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718963    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718966    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718968    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718971    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718974    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718977    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718979    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718983    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718985    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718988    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718991    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718993    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:44.719664 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718997    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.718999    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719002    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719004    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719007    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719010    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719013    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719016    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719018    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719021    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719023    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719026    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719029    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719032    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719037    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719040    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719042    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719045    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719047    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719050    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:44.720175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719052    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719055    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719057    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719060    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719062    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719065    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719067    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719070    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719073    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719076    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719078    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719082    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719086    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719089    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719092    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719094    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719098    2562 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719101    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719104    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:44.720728 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719106    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719109    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719112    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719114    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719118    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719120    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719123    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719125    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719128    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719131    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719133    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719136    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719139    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719141    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719143    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719146    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719149    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719151    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719154    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:44.721182 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719156    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.719162    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719272    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719278    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719281    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719284    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719287    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719290    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719293    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719296    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719299    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719303    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719305    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719308    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719310    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:44.721693 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719313    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719316    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719319    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719321    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719324    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719326    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719329    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719332    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719334    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719337    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719339    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719342    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719345    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719347    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719350    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719352    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719355    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719357    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719359    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719362    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:44.722061 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719364    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719367    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719370    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719373    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719375    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719378    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719380    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719383    2562 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719386    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719389    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719391    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719394    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719397    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719399    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719403    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719406    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719409    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719411    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719414    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:44.722567 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719417    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719419    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719422    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719426    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719429    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719433    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719436    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719438    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719441    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719444    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719447    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719449    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719452    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719455    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719457    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719463    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719465    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719468    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719471    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719473    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:44.723025 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719476    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719479    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719481    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719484    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719487    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719489    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719492    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719494    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719497    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719499    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719502    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719505    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719507    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:44.719510    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.719515    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:12:44.723625 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.720274    2562 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 12:12:44.724146 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.723377    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 12:12:44.724219 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.724206    2562 server.go:1019] "Starting client certificate rotation"
May 25 12:12:44.724310 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.724292    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:12:44.724370 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.724341    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:12:44.749851 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.749830    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:12:44.752226 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.752196    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:12:44.767975 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.767949    2562 log.go:25] "Validated CRI v1 runtime API"
May 25 12:12:44.774614 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.774598    2562 log.go:25] "Validated CRI v1 image API"
May 25 12:12:44.776058 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.776040    2562 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 12:12:44.778903 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.778883    2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:12:44.781123 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.781099    2562 fs.go:135] Filesystem UUIDs: map[5d861646-9f97-43d7-8c59-a377fbdf479e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dbbdadba-e659-414d-9f8a-a35f0889e550:/dev/nvme0n1p3]
May 25 12:12:44.781167 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.781119    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 12:12:44.785365 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.785235    2562 manager.go:217] Machine: {Timestamp:2026-05-25 12:12:44.784742489 +0000 UTC m=+0.443073887 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099546 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2661f7b651b46886d8a305940f90c2 SystemUUID:ec2661f7-b651-b468-86d8-a305940f90c2 BootID:ba4c7ba4-2dee-4964-8c86-eef386c2baee Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:78:5d:83:2f:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:78:5d:83:2f:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:ae:aa:30:8e:a3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 12:12:44.786046 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.786035    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 12:12:44.786172 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.786160    2562 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 12:12:44.787849 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.787824    2562 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 12:12:44.787993 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.787852    2562 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 12:12:44.788039 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.788000    2562 topology_manager.go:138] "Creating topology manager with none policy"
May 25 12:12:44.788039 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.788009    2562 container_manager_linux.go:306] "Creating device plugin manager"
May 25 12:12:44.788039 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.788016    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:12:44.788791 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.788781    2562 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:12:44.789675 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.789665    2562 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:12:44.789780 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.789772    2562 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 12:12:44.792051 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.792041    2562 kubelet.go:491] "Attempting to sync node with API server"
May 25 12:12:44.792089 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.792077    2562 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 12:12:44.792131 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.792089    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 12:12:44.792131 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.792100    2562 kubelet.go:397] "Adding apiserver pod source"
May 25 12:12:44.792131 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.792109    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 12:12:44.793127 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.793115    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:12:44.793173 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.793134    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:12:44.796767 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.796751    2562 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 12:12:44.801864 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.801843    2562 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 12:12:44.803396 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803375    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 12:12:44.803396 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803397    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803404    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803409    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803418    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803424    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803429    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803435    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803442    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803448    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803466    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 12:12:44.803498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.803472    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 12:12:44.806679 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.806660    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 12:12:44.806741 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.806681    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 12:12:44.807224 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.807202    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 12:12:44.807282 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.807201    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 12:12:44.809975 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.809843    2562 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 12:12:44.810059 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.809999    2562 server.go:1295] "Started kubelet"
May 25 12:12:44.810102 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.810063    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 12:12:44.810140 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.810085    2562 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 12:12:44.810169 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.810148    2562 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 12:12:44.810865 ip-10-0-141-91 systemd[1]: Started Kubernetes Kubelet.
May 25 12:12:44.811895 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.811873    2562 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 12:12:44.812898 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.812883    2562 server.go:317] "Adding debug handlers to kubelet server"
May 25 12:12:44.816947 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.816930    2562 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 12:12:44.817361 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.817347    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 12:12:44.817942 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.817928    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 12:12:44.818113 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.818102    2562 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 12:12:44.818165 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.818114    2562 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 12:12:44.818233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.818223    2562 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 12:12:44.818306 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.816995    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-91.ec2.internal.18b2cde0054b922e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-91.ec2.internal,UID:ip-10-0-141-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-91.ec2.internal,},FirstTimestamp:2026-05-25 12:12:44.809974318 +0000 UTC m=+0.468305714,LastTimestamp:2026-05-25 12:12:44.809974318 +0000 UTC m=+0.468305714,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-91.ec2.internal,}"
May 25 12:12:44.818306 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.818306    2562 reconstruct.go:97] "Volume reconstruction finished"
May 25 12:12:44.818402 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.818314    2562 reconciler.go:26] "Reconciler: start to sync state"
May 25 12:12:44.820063 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.818980    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:44.822455 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.822439    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 12:12:44.822455 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.822457    2562 factory.go:55] Registering systemd factory
May 25 12:12:44.822607 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.822463    2562 factory.go:223] Registration of the systemd container factory successfully
May 25 12:12:44.822720 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.822706    2562 factory.go:153] Registering CRI-O factory
May 25 12:12:44.822720 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.822721    2562 factory.go:223] Registration of the crio container factory successfully
May 25 12:12:44.822840 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.822737    2562 factory.go:103] Registering Raw factory
May 25 12:12:44.822840 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.822752    2562 manager.go:1196] Started watching for new ooms in manager
May 25 12:12:44.823141 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.823126    2562 manager.go:319] Starting recovery of all containers
May 25 12:12:44.825124 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.825094    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rm7pq"
May 25 12:12:44.826415 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.826390    2562 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 12:12:44.827936 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.827907    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 12:12:44.828024 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.827926    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 12:12:44.832610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.832579    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rm7pq"
May 25 12:12:44.834564 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.834550    2562 manager.go:324] Recovery completed
May 25 12:12:44.838429 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.838417    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:44.840281 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.840247    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:44.840364 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.840289    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:44.840364 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.840299    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:44.840837 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.840820    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 12:12:44.840837 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.840835    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 12:12:44.840978 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.840850    2562 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:12:44.843150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.843137    2562 policy_none.go:49] "None policy: Start"
May 25 12:12:44.843150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.843153    2562 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 12:12:44.843248 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.843164    2562 state_mem.go:35] "Initializing new in-memory state store"
May 25 12:12:44.843372 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.841720    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-91.ec2.internal.18b2cde00719f52f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-91.ec2.internal,UID:ip-10-0-141-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-91.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-91.ec2.internal,},FirstTimestamp:2026-05-25 12:12:44.840277295 +0000 UTC m=+0.498608691,LastTimestamp:2026-05-25 12:12:44.840277295 +0000 UTC m=+0.498608691,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-91.ec2.internal,}"
May 25 12:12:44.874846 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.874830    2562 manager.go:341] "Starting Device Plugin manager"
May 25 12:12:44.886608 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.874905    2562 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 12:12:44.886608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.874918    2562 server.go:85] "Starting device plugin registration server"
May 25 12:12:44.886608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.875154    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 12:12:44.886608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.875168    2562 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 12:12:44.886608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.875277    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 12:12:44.886608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.875365    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 12:12:44.886608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.875372    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 12:12:44.886608 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.875778    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 12:12:44.886608 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.875811    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:44.948739 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.948708    2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 12:12:44.949908 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.949890    2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 12:12:44.949976 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.949925    2562 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 12:12:44.949976 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.949951    2562 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 12:12:44.949976 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.949959    2562 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 12:12:44.950084 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.950005    2562 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 12:12:44.952551 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.952503    2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:44.975779 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.975762    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:44.976624 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.976609    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:44.976680 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.976637    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:44.976680 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.976648    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:44.976680 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.976672    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-91.ec2.internal"
May 25 12:12:44.984963 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:44.984946    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-91.ec2.internal"
May 25 12:12:44.985009 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:44.984968    2562 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-91.ec2.internal\": node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.002385 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.002363    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.050767 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.050734    2562 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal"]
May 25 12:12:45.050891 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.050816    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:45.052519 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.052504    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:45.052582 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.052534    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:45.052582 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.052545    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:45.053927 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.053915    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:45.054062 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.054049    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.054099 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.054077    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:45.054656 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.054637    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:45.054656 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.054650    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:45.054798 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.054668    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:45.054798 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.054676    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:45.054798 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.054680    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:45.054798 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.054686    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:45.055896 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.055881    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.055950 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.055904    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:45.056574 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.056557    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:45.056656 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.056585    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:45.056656 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.056602    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:45.084174 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.084152    2562 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-91.ec2.internal\" not found" node="ip-10-0-141-91.ec2.internal"
May 25 12:12:45.088731 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.088712    2562 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-91.ec2.internal\" not found" node="ip-10-0-141-91.ec2.internal"
May 25 12:12:45.103270 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.103236    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.204111 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.204044    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.220369 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.220350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0629a377029402e8a8f4f083a090ea51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal\" (UID: \"0629a377029402e8a8f4f083a090ea51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.220436 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.220373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0629a377029402e8a8f4f083a090ea51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal\" (UID: \"0629a377029402e8a8f4f083a090ea51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.220436 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.220392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e4a54b448df6387855e6ddf700b60d5-config\") pod \"kube-apiserver-proxy-ip-10-0-141-91.ec2.internal\" (UID: \"8e4a54b448df6387855e6ddf700b60d5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.304778 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.304737    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.321113 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.321090    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0629a377029402e8a8f4f083a090ea51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal\" (UID: \"0629a377029402e8a8f4f083a090ea51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.321167 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.321121    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0629a377029402e8a8f4f083a090ea51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal\" (UID: \"0629a377029402e8a8f4f083a090ea51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.321167 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.321139    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e4a54b448df6387855e6ddf700b60d5-config\") pod \"kube-apiserver-proxy-ip-10-0-141-91.ec2.internal\" (UID: \"8e4a54b448df6387855e6ddf700b60d5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.321230 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.321189    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0629a377029402e8a8f4f083a090ea51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal\" (UID: \"0629a377029402e8a8f4f083a090ea51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.321230 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.321200    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8e4a54b448df6387855e6ddf700b60d5-config\") pod \"kube-apiserver-proxy-ip-10-0-141-91.ec2.internal\" (UID: \"8e4a54b448df6387855e6ddf700b60d5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.321308 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.321233    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0629a377029402e8a8f4f083a090ea51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal\" (UID: \"0629a377029402e8a8f4f083a090ea51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.387239 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.387215    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.391835 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.391817    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.405585 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.405564    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.506123 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.506041    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.606536 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.606503    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.707122 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.707089    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.724509 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.724485    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 12:12:45.724627 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.724613    2562 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:12:45.808156 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:45.808121    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-91.ec2.internal\" not found"
May 25 12:12:45.817697 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.817668    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 12:12:45.832326 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.832305    2562 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:45.834286 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.834241    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 12:07:44 +0000 UTC" deadline="2028-02-02 14:46:16.690576728 +0000 UTC"
May 25 12:12:45.834286 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.834283    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14834h33m30.856296803s"
May 25 12:12:45.838067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.838049    2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:12:45.871524 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.871494    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9qncs"
May 25 12:12:45.874114 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:45.874080    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0629a377029402e8a8f4f083a090ea51.slice/crio-4301f929cc82192d2947b5e281a4446334237fd37f36f2cdaedb59cf44eea95b WatchSource:0}: Error finding container 4301f929cc82192d2947b5e281a4446334237fd37f36f2cdaedb59cf44eea95b: Status 404 returned error can't find the container with id 4301f929cc82192d2947b5e281a4446334237fd37f36f2cdaedb59cf44eea95b
May 25 12:12:45.874375 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:45.874357    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4a54b448df6387855e6ddf700b60d5.slice/crio-19af8fd940baaaa90af9351515f4a1d72122febf600f508613c8bf6ca0ab4446 WatchSource:0}: Error finding container 19af8fd940baaaa90af9351515f4a1d72122febf600f508613c8bf6ca0ab4446: Status 404 returned error can't find the container with id 19af8fd940baaaa90af9351515f4a1d72122febf600f508613c8bf6ca0ab4446
May 25 12:12:45.876664 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.876639    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9qncs"
May 25 12:12:45.877659 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.877642    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:12:45.918635 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.918605    2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.927721 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.927702    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:12:45.929490 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.929479    2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal"
May 25 12:12:45.939644 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.939630    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:12:45.952732 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.952691    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal" event={"ID":"0629a377029402e8a8f4f083a090ea51","Type":"ContainerStarted","Data":"4301f929cc82192d2947b5e281a4446334237fd37f36f2cdaedb59cf44eea95b"}
May 25 12:12:45.953620 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:45.953590    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal" event={"ID":"8e4a54b448df6387855e6ddf700b60d5","Type":"ContainerStarted","Data":"19af8fd940baaaa90af9351515f4a1d72122febf600f508613c8bf6ca0ab4446"}
May 25 12:12:46.005727 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.005703    2562 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:46.157271 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.157182    2562 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:46.793491 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.793446    2562 apiserver.go:52] "Watching apiserver"
May 25 12:12:46.800450 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.800410    2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:46.801982 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.801955    2562 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 12:12:46.802399 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.802372    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-22jr7","openshift-ovn-kubernetes/ovnkube-node-xrrrl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs","openshift-dns/node-resolver-j9q65","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal","openshift-multus/multus-7hkxn","openshift-network-diagnostics/network-check-target-rvhv8","openshift-network-operator/iptables-alerter-2d962","kube-system/konnectivity-agent-dg8fc","kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal","openshift-cluster-node-tuning-operator/tuned-tmtnm","openshift-image-registry/node-ca-cf9x4","openshift-multus/multus-additional-cni-plugins-7sfdq"]
May 25 12:12:46.804152 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.804122    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:12:46.806673 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.806642    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.807009 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.806811    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 12:12:46.807009 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.806874    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5h6bn\""
May 25 12:12:46.807187 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.807032    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 12:12:46.807955 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.807917    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:46.808060 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.808002    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.808060 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:46.808004    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:12:46.809128 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.809104    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 12:12:46.809544 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.809362    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 12:12:46.809544 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.809440    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 12:12:46.809857 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.809834    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 12:12:46.810168 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.810151    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9g6m\""
May 25 12:12:46.810342 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.810224    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 12:12:46.811020 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.810490    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 12:12:46.811020 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.810694    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 12:12:46.811020 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.810810    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4n65r\""
May 25 12:12:46.813719 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.813382    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.813719 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.813490    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:46.813719 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:46.813615    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:12:46.814830 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.814769    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.815918 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.815888    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 12:12:46.816046 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.815888    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8bc9b\""
May 25 12:12:46.817159 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.817140    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 12:12:46.817496 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.817476    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f97m4\""
May 25 12:12:46.817596 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.817530    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 12:12:46.817682 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.817661    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 12:12:46.817775 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.817732    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.818931 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.818890    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.819626 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.819142    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.820089 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.820071    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nkhm5\""
May 25 12:12:46.820338 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.820320    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 12:12:46.820431 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.820338    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 12:12:46.820431 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.820371    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 12:12:46.820720 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.820698    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.821306 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.821283    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 12:12:46.821512 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.821493    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 12:12:46.821690 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.821676    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jjgjh\""
May 25 12:12:46.821753 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.821708    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 12:12:46.822044 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.822026    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 12:12:46.822126 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.822082    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2wxcc\""
May 25 12:12:46.822182 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.822165    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 12:12:46.823184 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.823060    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 12:12:46.823458 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.823411    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 12:12:46.823695 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.823679    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 12:12:46.824391 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.824372    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 12:12:46.824481 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.824392    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 12:12:46.824808 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.824789    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4rw66\""
May 25 12:12:46.824883 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.824841    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 12:12:46.828363 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-system-cni-dir\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.828468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/782e7027-de68-4d9f-83de-04b38965cd87-iptables-alerter-script\") pod \"iptables-alerter-2d962\" (UID: \"782e7027-de68-4d9f-83de-04b38965cd87\") " pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.828468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828412    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-sysctl-d\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.828468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-run\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.828468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-var-lib-kubelet\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.828642 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f-serviceca\") pod \"node-ca-cf9x4\" (UID: \"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f\") " pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.828642 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-systemd-units\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.828642 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828527    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-run-systemd\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.828642 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828550    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qvg\" (UniqueName: \"kubernetes.io/projected/782e7027-de68-4d9f-83de-04b38965cd87-kube-api-access-s7qvg\") pod \"iptables-alerter-2d962\" (UID: \"782e7027-de68-4d9f-83de-04b38965cd87\") " pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.828642 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-sys-fs\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.828642 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67kj\" (UniqueName: \"kubernetes.io/projected/77f68379-dc11-4aaf-9179-daf22dfddab1-kube-api-access-j67kj\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.828642 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-system-cni-dir\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.828642 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-var-lib-openvswitch\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.828869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-etc-openvswitch\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.828869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-node-log\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.828869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-log-socket\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.828869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-cnibin\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.828869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jp8p\" (UniqueName: \"kubernetes.io/projected/037bfb47-48b6-4248-9537-b2da35bdf80f-kube-api-access-8jp8p\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:46.828869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-systemd\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.828869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krpg\" (UniqueName: \"kubernetes.io/projected/249e00b6-ef97-49ed-8be8-069779eace7d-kube-api-access-7krpg\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.828869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-var-lib-cni-bin\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.828869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-run-openvswitch\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-cni-netd\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96d029b0-6c0f-496a-b1f0-5a651825fc29-env-overrides\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-socket-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-socket-dir-parent\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.828990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-run-netns\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-run-multus-certs\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7g8\" (UniqueName: \"kubernetes.io/projected/cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f-kube-api-access-4n7g8\") pod \"node-ca-cf9x4\" (UID: \"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f\") " pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829072    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96d029b0-6c0f-496a-b1f0-5a651825fc29-ovn-node-metrics-cert\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-sysconfig\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/249e00b6-ef97-49ed-8be8-069779eace7d-etc-tuned\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.829194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/994d3f70-8d53-4232-9eec-f97966e0fa11-cni-binary-copy\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-run-k8s-cni-cncf-io\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-conf-dir\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01253226-6d29-4eac-9a58-3d9a8be0032d-tmp-dir\") pod \"node-resolver-j9q65\" (UID: \"01253226-6d29-4eac-9a58-3d9a8be0032d\") " pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/128d93e5-705b-4d8f-b4b0-60af82c40726-agent-certs\") pod \"konnectivity-agent-dg8fc\" (UID: \"128d93e5-705b-4d8f-b4b0-60af82c40726\") " pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-cni-dir\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-slash\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-modprobe-d\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829593    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f-host\") pod \"node-ca-cf9x4\" (UID: \"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f\") " pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829615    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-kubelet\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9t6\" (UniqueName: \"kubernetes.io/projected/96d029b0-6c0f-496a-b1f0-5a651825fc29-kube-api-access-vx9t6\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-sys\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-var-lib-cni-multus\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-run-netns\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-run-ovn\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-os-release\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.829872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-etc-selinux\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wl6g\" (UniqueName: \"kubernetes.io/projected/994d3f70-8d53-4232-9eec-f97966e0fa11-kube-api-access-2wl6g\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829910    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96d029b0-6c0f-496a-b1f0-5a651825fc29-ovnkube-config\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96d029b0-6c0f-496a-b1f0-5a651825fc29-ovnkube-script-lib\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-device-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.829996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-kubernetes\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-daemon-config\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830051    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcdr\" (UniqueName: \"kubernetes.io/projected/e1367fb0-fb43-4995-9af7-9be8baf478af-kube-api-access-zkcdr\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-cnibin\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01253226-6d29-4eac-9a58-3d9a8be0032d-hosts-file\") pod \"node-resolver-j9q65\" (UID: \"01253226-6d29-4eac-9a58-3d9a8be0032d\") " pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1367fb0-fb43-4995-9af7-9be8baf478af-cni-binary-copy\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830176    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-lib-modules\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-host\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-cni-bin\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.830610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1367fb0-fb43-4995-9af7-9be8baf478af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-hostroot\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/782e7027-de68-4d9f-83de-04b38965cd87-host-slash\") pod \"iptables-alerter-2d962\" (UID: \"782e7027-de68-4d9f-83de-04b38965cd87\") " pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830463    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-registration-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-sysctl-conf\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830513    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/128d93e5-705b-4d8f-b4b0-60af82c40726-konnectivity-ca\") pod \"konnectivity-agent-dg8fc\" (UID: \"128d93e5-705b-4d8f-b4b0-60af82c40726\") " pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1367fb0-fb43-4995-9af7-9be8baf478af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbf9t\" (UniqueName: \"kubernetes.io/projected/01253226-6d29-4eac-9a58-3d9a8be0032d-kube-api-access-mbf9t\") pod \"node-resolver-j9q65\" (UID: \"01253226-6d29-4eac-9a58-3d9a8be0032d\") " pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-var-lib-kubelet\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/249e00b6-ef97-49ed-8be8-069779eace7d-tmp\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-os-release\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.831349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.830640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-etc-kubernetes\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.877643 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.877606    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:07:45 +0000 UTC" deadline="2027-12-12 06:51:48.20556602 +0000 UTC"
May 25 12:12:46.877643 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.877630    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13578h39m1.327938247s"
May 25 12:12:46.919705 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.919671    2562 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 12:12:46.931787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931759    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-device-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.931787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931791    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-kubernetes\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931811    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-daemon-config\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931835    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcdr\" (UniqueName: \"kubernetes.io/projected/e1367fb0-fb43-4995-9af7-9be8baf478af-kube-api-access-zkcdr\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931861    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-cnibin\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931877    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-device-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931888    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01253226-6d29-4eac-9a58-3d9a8be0032d-hosts-file\") pod \"node-resolver-j9q65\" (UID: \"01253226-6d29-4eac-9a58-3d9a8be0032d\") " pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931894    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-kubernetes\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931911    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1367fb0-fb43-4995-9af7-9be8baf478af-cni-binary-copy\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931942    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-cnibin\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931947    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01253226-6d29-4eac-9a58-3d9a8be0032d-hosts-file\") pod \"node-resolver-j9q65\" (UID: \"01253226-6d29-4eac-9a58-3d9a8be0032d\") " pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.931965    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.932011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932011    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-lib-modules\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932040    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-host\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932063    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-cni-bin\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932087    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932138    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-host\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932088    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1367fb0-fb43-4995-9af7-9be8baf478af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932178    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-cni-bin\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932169    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-lib-modules\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932212    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-hostroot\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932237    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932273    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/782e7027-de68-4d9f-83de-04b38965cd87-host-slash\") pod \"iptables-alerter-2d962\" (UID: \"782e7027-de68-4d9f-83de-04b38965cd87\") " pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932300    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:46.932357    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932366    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-registration-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932386    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932390    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-sysctl-conf\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932299    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-hostroot\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.932530 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932330    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/782e7027-de68-4d9f-83de-04b38965cd87-host-slash\") pod \"iptables-alerter-2d962\" (UID: \"782e7027-de68-4d9f-83de-04b38965cd87\") " pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:46.932470    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs podName:037bfb47-48b6-4248-9537-b2da35bdf80f nodeName:}" failed. No retries permitted until 2026-05-25 12:12:47.432416498 +0000 UTC m=+3.090747898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs") pod "network-metrics-daemon-22jr7" (UID: "037bfb47-48b6-4248-9537-b2da35bdf80f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932477    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-registration-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932493    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/128d93e5-705b-4d8f-b4b0-60af82c40726-konnectivity-ca\") pod \"konnectivity-agent-dg8fc\" (UID: \"128d93e5-705b-4d8f-b4b0-60af82c40726\") " pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932520    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-daemon-config\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932536    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1367fb0-fb43-4995-9af7-9be8baf478af-cni-binary-copy\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932563    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1367fb0-fb43-4995-9af7-9be8baf478af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932524    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1367fb0-fb43-4995-9af7-9be8baf478af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932568    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-sysctl-conf\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932608    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mbf9t\" (UniqueName: \"kubernetes.io/projected/01253226-6d29-4eac-9a58-3d9a8be0032d-kube-api-access-mbf9t\") pod \"node-resolver-j9q65\" (UID: \"01253226-6d29-4eac-9a58-3d9a8be0032d\") " pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932634    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-var-lib-kubelet\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932657    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/249e00b6-ef97-49ed-8be8-069779eace7d-tmp\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932680    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-os-release\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932703    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-etc-kubernetes\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932735    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-var-lib-kubelet\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932752    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-system-cni-dir\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.933498 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932778    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/782e7027-de68-4d9f-83de-04b38965cd87-iptables-alerter-script\") pod \"iptables-alerter-2d962\" (UID: \"782e7027-de68-4d9f-83de-04b38965cd87\") " pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932802    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-sysctl-d\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932826    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-run\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932832    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-os-release\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932849    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-var-lib-kubelet\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932878    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f-serviceca\") pod \"node-ca-cf9x4\" (UID: \"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f\") " pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932901    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-systemd-units\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932906    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-etc-kubernetes\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932925    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-run-systemd\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932951    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qvg\" (UniqueName: \"kubernetes.io/projected/782e7027-de68-4d9f-83de-04b38965cd87-kube-api-access-s7qvg\") pod \"iptables-alerter-2d962\" (UID: \"782e7027-de68-4d9f-83de-04b38965cd87\") " pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932955    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-var-lib-kubelet\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932989    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-sys-fs\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.932984    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933020    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j67kj\" (UniqueName: \"kubernetes.io/projected/77f68379-dc11-4aaf-9179-daf22dfddab1-kube-api-access-j67kj\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933050    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-system-cni-dir\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933068    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-sysctl-d\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933077    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-var-lib-openvswitch\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933080    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/128d93e5-705b-4d8f-b4b0-60af82c40726-konnectivity-ca\") pod \"konnectivity-agent-dg8fc\" (UID: \"128d93e5-705b-4d8f-b4b0-60af82c40726\") " pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:12:46.934233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933020    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1367fb0-fb43-4995-9af7-9be8baf478af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933104    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-etc-openvswitch\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933156    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-node-log\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933164    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-system-cni-dir\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933179    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-log-socket\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933129    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-run\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933228    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-var-lib-openvswitch\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933233    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-cnibin\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933236    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-node-log\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933287    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8jp8p\" (UniqueName: \"kubernetes.io/projected/037bfb47-48b6-4248-9537-b2da35bdf80f-kube-api-access-8jp8p\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933291    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-etc-openvswitch\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933297    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-log-socket\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933325    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-systemd-units\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933347    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-system-cni-dir\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933352    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-systemd\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933374    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f-serviceca\") pod \"node-ca-cf9x4\" (UID: \"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f\") " pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933380    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7krpg\" (UniqueName: \"kubernetes.io/projected/249e00b6-ef97-49ed-8be8-069779eace7d-kube-api-access-7krpg\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933384    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-run-systemd\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933355    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-sys-fs\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933426    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-cnibin\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933412    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-var-lib-cni-bin\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933481    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-systemd\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933527    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-var-lib-cni-bin\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933527    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-run-openvswitch\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933569    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-run-openvswitch\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933578    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/782e7027-de68-4d9f-83de-04b38965cd87-iptables-alerter-script\") pod \"iptables-alerter-2d962\" (UID: \"782e7027-de68-4d9f-83de-04b38965cd87\") " pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933601    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-cni-netd\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933633    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96d029b0-6c0f-496a-b1f0-5a651825fc29-env-overrides\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933664    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-socket-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933641    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-cni-netd\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933682    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-socket-dir-parent\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933717    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-run-netns\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933744    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-run-multus-certs\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933749    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-socket-dir-parent\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933775    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7g8\" (UniqueName: \"kubernetes.io/projected/cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f-kube-api-access-4n7g8\") pod \"node-ca-cf9x4\" (UID: \"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f\") " pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.935874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933782    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-socket-dir\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933796    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-run-multus-certs\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933803    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933832    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96d029b0-6c0f-496a-b1f0-5a651825fc29-ovn-node-metrics-cert\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933851    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933857    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-sysconfig\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933797    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-run-netns\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933882    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/249e00b6-ef97-49ed-8be8-069779eace7d-etc-tuned\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933906    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/994d3f70-8d53-4232-9eec-f97966e0fa11-cni-binary-copy\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933929    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-sysconfig\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.933974    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-run-k8s-cni-cncf-io\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934028    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-conf-dir\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934063    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934092    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01253226-6d29-4eac-9a58-3d9a8be0032d-tmp-dir\") pod \"node-resolver-j9q65\" (UID: \"01253226-6d29-4eac-9a58-3d9a8be0032d\") " pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934110    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96d029b0-6c0f-496a-b1f0-5a651825fc29-env-overrides\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934234    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-run-k8s-cni-cncf-io\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934117    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/128d93e5-705b-4d8f-b4b0-60af82c40726-agent-certs\") pod \"konnectivity-agent-dg8fc\" (UID: \"128d93e5-705b-4d8f-b4b0-60af82c40726\") " pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:12:46.936660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934348    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-cni-dir\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934374    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-slash\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934399    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-modprobe-d\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934422    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f-host\") pod \"node-ca-cf9x4\" (UID: \"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f\") " pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934457    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-kubelet\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934510    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9t6\" (UniqueName: \"kubernetes.io/projected/96d029b0-6c0f-496a-b1f0-5a651825fc29-kube-api-access-vx9t6\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934485    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-slash\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934534    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-sys\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934518    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01253226-6d29-4eac-9a58-3d9a8be0032d-tmp-dir\") pod \"node-resolver-j9q65\" (UID: \"01253226-6d29-4eac-9a58-3d9a8be0032d\") " pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934560    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-var-lib-cni-multus\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934564    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f-host\") pod \"node-ca-cf9x4\" (UID: \"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f\") " pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934572    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/994d3f70-8d53-4232-9eec-f97966e0fa11-cni-binary-copy\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934586    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-run-netns\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934621    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-kubelet\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934631    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-cni-dir\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934621    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-run-ovn\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934672    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-os-release\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934685    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-run-netns\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934673    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-host-var-lib-cni-multus\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934704    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-etc-selinux\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934731    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2wl6g\" (UniqueName: \"kubernetes.io/projected/994d3f70-8d53-4232-9eec-f97966e0fa11-kube-api-access-2wl6g\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934753    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1367fb0-fb43-4995-9af7-9be8baf478af-os-release\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934757    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934675    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-etc-modprobe-d\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934785    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96d029b0-6c0f-496a-b1f0-5a651825fc29-ovnkube-config\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934802    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/994d3f70-8d53-4232-9eec-f97966e0fa11-multus-conf-dir\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934811    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96d029b0-6c0f-496a-b1f0-5a651825fc29-ovnkube-script-lib\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934822    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77f68379-dc11-4aaf-9179-daf22dfddab1-etc-selinux\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934706    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-run-ovn\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934845    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96d029b0-6c0f-496a-b1f0-5a651825fc29-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.934754    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/249e00b6-ef97-49ed-8be8-069779eace7d-sys\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.935357    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96d029b0-6c0f-496a-b1f0-5a651825fc29-ovnkube-config\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.935429    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96d029b0-6c0f-496a-b1f0-5a651825fc29-ovnkube-script-lib\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.936328    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/249e00b6-ef97-49ed-8be8-069779eace7d-tmp\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.936712    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/249e00b6-ef97-49ed-8be8-069779eace7d-etc-tuned\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.936889    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96d029b0-6c0f-496a-b1f0-5a651825fc29-ovn-node-metrics-cert\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.937787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.937113    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/128d93e5-705b-4d8f-b4b0-60af82c40726-agent-certs\") pod \"konnectivity-agent-dg8fc\" (UID: \"128d93e5-705b-4d8f-b4b0-60af82c40726\") " pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:12:46.939871 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:46.939545    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:46.939871 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:46.939583    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:46.939871 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:46.939597    2562 projected.go:194] Error preparing data for projected volume kube-api-access-h2dj7 for pod openshift-network-diagnostics/network-check-target-rvhv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:46.939871 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:46.939660    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7 podName:0a521353-c135-45c9-a6f1-0d36c801505d nodeName:}" failed. No retries permitted until 2026-05-25 12:12:47.439641018 +0000 UTC m=+3.097972421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h2dj7" (UniqueName: "kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7") pod "network-check-target-rvhv8" (UID: "0a521353-c135-45c9-a6f1-0d36c801505d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:46.942840 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.941819    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcdr\" (UniqueName: \"kubernetes.io/projected/e1367fb0-fb43-4995-9af7-9be8baf478af-kube-api-access-zkcdr\") pod \"multus-additional-cni-plugins-7sfdq\" (UID: \"e1367fb0-fb43-4995-9af7-9be8baf478af\") " pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:46.942840 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.942247    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbf9t\" (UniqueName: \"kubernetes.io/projected/01253226-6d29-4eac-9a58-3d9a8be0032d-kube-api-access-mbf9t\") pod \"node-resolver-j9q65\" (UID: \"01253226-6d29-4eac-9a58-3d9a8be0032d\") " pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:46.942840 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.942411    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7g8\" (UniqueName: \"kubernetes.io/projected/cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f-kube-api-access-4n7g8\") pod \"node-ca-cf9x4\" (UID: \"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f\") " pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:46.942840 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.942776    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qvg\" (UniqueName: \"kubernetes.io/projected/782e7027-de68-4d9f-83de-04b38965cd87-kube-api-access-s7qvg\") pod \"iptables-alerter-2d962\" (UID: \"782e7027-de68-4d9f-83de-04b38965cd87\") " pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:46.943153 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.943017    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9t6\" (UniqueName: \"kubernetes.io/projected/96d029b0-6c0f-496a-b1f0-5a651825fc29-kube-api-access-vx9t6\") pod \"ovnkube-node-xrrrl\" (UID: \"96d029b0-6c0f-496a-b1f0-5a651825fc29\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:46.943356 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.943333    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67kj\" (UniqueName: \"kubernetes.io/projected/77f68379-dc11-4aaf-9179-daf22dfddab1-kube-api-access-j67kj\") pod \"aws-ebs-csi-driver-node-kgdfs\" (UID: \"77f68379-dc11-4aaf-9179-daf22dfddab1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:46.944224 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.944202    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krpg\" (UniqueName: \"kubernetes.io/projected/249e00b6-ef97-49ed-8be8-069779eace7d-kube-api-access-7krpg\") pod \"tuned-tmtnm\" (UID: \"249e00b6-ef97-49ed-8be8-069779eace7d\") " pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:46.945543 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.945513    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wl6g\" (UniqueName: \"kubernetes.io/projected/994d3f70-8d53-4232-9eec-f97966e0fa11-kube-api-access-2wl6g\") pod \"multus-7hkxn\" (UID: \"994d3f70-8d53-4232-9eec-f97966e0fa11\") " pod="openshift-multus/multus-7hkxn"
May 25 12:12:46.945692 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:46.945601    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jp8p\" (UniqueName: \"kubernetes.io/projected/037bfb47-48b6-4248-9537-b2da35bdf80f-kube-api-access-8jp8p\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:47.116929 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.116849    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:12:47.125782 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.125752    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7sfdq"
May 25 12:12:47.133561 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.133535    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9q65"
May 25 12:12:47.139325 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.139297    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7hkxn"
May 25 12:12:47.146966 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.146935    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2d962"
May 25 12:12:47.153680 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.153655    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs"
May 25 12:12:47.161459 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.161432    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tmtnm"
May 25 12:12:47.168180 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.168151    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cf9x4"
May 25 12:12:47.174047 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.174019    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:12:47.438435 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.438357    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:47.438580 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.438474    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:47.438580 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.438527    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs podName:037bfb47-48b6-4248-9537-b2da35bdf80f nodeName:}" failed. No retries permitted until 2026-05-25 12:12:48.438511908 +0000 UTC m=+4.096843292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs") pod "network-metrics-daemon-22jr7" (UID: "037bfb47-48b6-4248-9537-b2da35bdf80f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:47.462573 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:47.462548    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1367fb0_fb43_4995_9af7_9be8baf478af.slice/crio-f770ff819781efb17288d52f9bd259cec2e7571eba13fc8a139d9e59c846fc89 WatchSource:0}: Error finding container f770ff819781efb17288d52f9bd259cec2e7571eba13fc8a139d9e59c846fc89: Status 404 returned error can't find the container with id f770ff819781efb17288d52f9bd259cec2e7571eba13fc8a139d9e59c846fc89
May 25 12:12:47.463574 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:47.463550    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d029b0_6c0f_496a_b1f0_5a651825fc29.slice/crio-55da364eb25a9537fb569be2e69bb2d30f7f968dafc343bf6b04db24c7dbe51e WatchSource:0}: Error finding container 55da364eb25a9537fb569be2e69bb2d30f7f968dafc343bf6b04db24c7dbe51e: Status 404 returned error can't find the container with id 55da364eb25a9537fb569be2e69bb2d30f7f968dafc343bf6b04db24c7dbe51e
May 25 12:12:47.464432 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:47.464285    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994d3f70_8d53_4232_9eec_f97966e0fa11.slice/crio-cb9020c37666e1327787b421b326efa01da124fcb2aa6b01096a08a075483aae WatchSource:0}: Error finding container cb9020c37666e1327787b421b326efa01da124fcb2aa6b01096a08a075483aae: Status 404 returned error can't find the container with id cb9020c37666e1327787b421b326efa01da124fcb2aa6b01096a08a075483aae
May 25 12:12:47.467595 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:47.467538    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f68379_dc11_4aaf_9179_daf22dfddab1.slice/crio-ec85a32cd066cfa19a29a0c014b46d45080728b303d26f1d322694e645473db7 WatchSource:0}: Error finding container ec85a32cd066cfa19a29a0c014b46d45080728b303d26f1d322694e645473db7: Status 404 returned error can't find the container with id ec85a32cd066cfa19a29a0c014b46d45080728b303d26f1d322694e645473db7
May 25 12:12:47.468398 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:47.468377    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249e00b6_ef97_49ed_8be8_069779eace7d.slice/crio-6456c34654b5bf37846aa82e7dca9643a5fb4e0cda7b9c3e7611575d7d50aaf9 WatchSource:0}: Error finding container 6456c34654b5bf37846aa82e7dca9643a5fb4e0cda7b9c3e7611575d7d50aaf9: Status 404 returned error can't find the container with id 6456c34654b5bf37846aa82e7dca9643a5fb4e0cda7b9c3e7611575d7d50aaf9
May 25 12:12:47.469175 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:47.469073    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb76c49a_e523_4f5b_a0ae_3ac0d8b0442f.slice/crio-c504f3becf319b80240bb60ca1f55649945e34b8389bd9d807264d22dc682b60 WatchSource:0}: Error finding container c504f3becf319b80240bb60ca1f55649945e34b8389bd9d807264d22dc682b60: Status 404 returned error can't find the container with id c504f3becf319b80240bb60ca1f55649945e34b8389bd9d807264d22dc682b60
May 25 12:12:47.469988 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:47.469763    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128d93e5_705b_4d8f_b4b0_60af82c40726.slice/crio-ec638ded1cdc6b87e15cc4274de5d1698774a1c3fd6b0855069920d04839d0d5 WatchSource:0}: Error finding container ec638ded1cdc6b87e15cc4274de5d1698774a1c3fd6b0855069920d04839d0d5: Status 404 returned error can't find the container with id ec638ded1cdc6b87e15cc4274de5d1698774a1c3fd6b0855069920d04839d0d5
May 25 12:12:47.471208 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:12:47.471088    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782e7027_de68_4d9f_83de_04b38965cd87.slice/crio-a36417eeb8b5f0b3d8f848922d70c9928572d08fc22c35b87afc0e9c4e89e478 WatchSource:0}: Error finding container a36417eeb8b5f0b3d8f848922d70c9928572d08fc22c35b87afc0e9c4e89e478: Status 404 returned error can't find the container with id a36417eeb8b5f0b3d8f848922d70c9928572d08fc22c35b87afc0e9c4e89e478
May 25 12:12:47.539544 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.539516    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:47.539717 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.539698    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:47.539768 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.539731    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:47.539768 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.539745    2562 projected.go:194] Error preparing data for projected volume kube-api-access-h2dj7 for pod openshift-network-diagnostics/network-check-target-rvhv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:47.539848 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.539805    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7 podName:0a521353-c135-45c9-a6f1-0d36c801505d nodeName:}" failed. No retries permitted until 2026-05-25 12:12:48.539787147 +0000 UTC m=+4.198118532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2dj7" (UniqueName: "kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7") pod "network-check-target-rvhv8" (UID: "0a521353-c135-45c9-a6f1-0d36c801505d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:47.771528 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.771502    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-wlsj7"]
May 25 12:12:47.773329 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.773309    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:47.773452 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.773402    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:12:47.842558 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.842517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f71c2f0-743b-4295-af1d-2182812eaeb6-dbus\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:47.843104 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.842572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f71c2f0-743b-4295-af1d-2182812eaeb6-kubelet-config\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:47.843104 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.842608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:47.878110 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.878060    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:07:45 +0000 UTC" deadline="2028-01-19 04:33:08.465890021 +0000 UTC"
May 25 12:12:47.878110 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.878107    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14488h20m20.587786647s"
May 25 12:12:47.948624 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.943480    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f71c2f0-743b-4295-af1d-2182812eaeb6-dbus\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:47.948624 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.943516    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f71c2f0-743b-4295-af1d-2182812eaeb6-kubelet-config\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:47.948624 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.943548    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:47.948624 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.943669    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:12:47.948624 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.943736    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret podName:3f71c2f0-743b-4295-af1d-2182812eaeb6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:48.443707616 +0000 UTC m=+4.102039002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret") pod "global-pull-secret-syncer-wlsj7" (UID: "3f71c2f0-743b-4295-af1d-2182812eaeb6") : object "kube-system"/"original-pull-secret" not registered
May 25 12:12:47.948624 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.944071    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f71c2f0-743b-4295-af1d-2182812eaeb6-dbus\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:47.948624 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.944139    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f71c2f0-743b-4295-af1d-2182812eaeb6-kubelet-config\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:47.950945 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.950710    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:47.950945 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:47.950836    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:12:47.976693 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.976584    2562 generic.go:358] "Generic (PLEG): container finished" podID="0629a377029402e8a8f4f083a090ea51" containerID="978922108a5cb4cafd4ea8e6952540509cddaedf204a41f4454525482155f6c3" exitCode=0
May 25 12:12:47.976856 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.976752    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal" event={"ID":"0629a377029402e8a8f4f083a090ea51","Type":"ContainerDied","Data":"978922108a5cb4cafd4ea8e6952540509cddaedf204a41f4454525482155f6c3"}
May 25 12:12:47.988872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:47.988834    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2d962" event={"ID":"782e7027-de68-4d9f-83de-04b38965cd87","Type":"ContainerStarted","Data":"a36417eeb8b5f0b3d8f848922d70c9928572d08fc22c35b87afc0e9c4e89e478"}
May 25 12:12:48.004937 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.004856    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dg8fc" event={"ID":"128d93e5-705b-4d8f-b4b0-60af82c40726","Type":"ContainerStarted","Data":"ec638ded1cdc6b87e15cc4274de5d1698774a1c3fd6b0855069920d04839d0d5"}
May 25 12:12:48.011675 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.011601    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tmtnm" event={"ID":"249e00b6-ef97-49ed-8be8-069779eace7d","Type":"ContainerStarted","Data":"6456c34654b5bf37846aa82e7dca9643a5fb4e0cda7b9c3e7611575d7d50aaf9"}
May 25 12:12:48.018852 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.018813    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs" event={"ID":"77f68379-dc11-4aaf-9179-daf22dfddab1","Type":"ContainerStarted","Data":"ec85a32cd066cfa19a29a0c014b46d45080728b303d26f1d322694e645473db7"}
May 25 12:12:48.023458 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.023427    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7hkxn" event={"ID":"994d3f70-8d53-4232-9eec-f97966e0fa11","Type":"ContainerStarted","Data":"cb9020c37666e1327787b421b326efa01da124fcb2aa6b01096a08a075483aae"}
May 25 12:12:48.029289 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.029232    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" event={"ID":"96d029b0-6c0f-496a-b1f0-5a651825fc29","Type":"ContainerStarted","Data":"55da364eb25a9537fb569be2e69bb2d30f7f968dafc343bf6b04db24c7dbe51e"}
May 25 12:12:48.041734 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.041643    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9q65" event={"ID":"01253226-6d29-4eac-9a58-3d9a8be0032d","Type":"ContainerStarted","Data":"34e8914c938c380ace1af95ea221896d6bc7477b4d95b3cca46bcae308e816c6"}
May 25 12:12:48.060579 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.060512    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal" event={"ID":"8e4a54b448df6387855e6ddf700b60d5","Type":"ContainerStarted","Data":"e5c050bc289f866df733338d43e6f2ddac9673fa011fd5306807d4938372bb51"}
May 25 12:12:48.074181 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.074102    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cf9x4" event={"ID":"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f","Type":"ContainerStarted","Data":"c504f3becf319b80240bb60ca1f55649945e34b8389bd9d807264d22dc682b60"}
May 25 12:12:48.084330 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.084199    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sfdq" event={"ID":"e1367fb0-fb43-4995-9af7-9be8baf478af","Type":"ContainerStarted","Data":"f770ff819781efb17288d52f9bd259cec2e7571eba13fc8a139d9e59c846fc89"}
May 25 12:12:48.448252 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.448168    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:48.448252 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.448223    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:48.448466 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.448403    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:48.448536 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.448468    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs podName:037bfb47-48b6-4248-9537-b2da35bdf80f nodeName:}" failed. No retries permitted until 2026-05-25 12:12:50.448447835 +0000 UTC m=+6.106779220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs") pod "network-metrics-daemon-22jr7" (UID: "037bfb47-48b6-4248-9537-b2da35bdf80f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:48.448879 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.448862    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:12:48.448973 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.448915    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret podName:3f71c2f0-743b-4295-af1d-2182812eaeb6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:49.448900105 +0000 UTC m=+5.107231488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret") pod "global-pull-secret-syncer-wlsj7" (UID: "3f71c2f0-743b-4295-af1d-2182812eaeb6") : object "kube-system"/"original-pull-secret" not registered
May 25 12:12:48.549111 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.549066    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:48.549343 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.549232    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:48.549343 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.549251    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:48.549343 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.549286    2562 projected.go:194] Error preparing data for projected volume kube-api-access-h2dj7 for pod openshift-network-diagnostics/network-check-target-rvhv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:48.549538 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.549386    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7 podName:0a521353-c135-45c9-a6f1-0d36c801505d nodeName:}" failed. No retries permitted until 2026-05-25 12:12:50.549359848 +0000 UTC m=+6.207691233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2dj7" (UniqueName: "kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7") pod "network-check-target-rvhv8" (UID: "0a521353-c135-45c9-a6f1-0d36c801505d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:48.953481 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.953445    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:48.953934 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.953592    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:12:48.953934 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:48.953698    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:48.953934 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:48.953777    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:12:49.102430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:49.102327    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal" event={"ID":"0629a377029402e8a8f4f083a090ea51","Type":"ContainerStarted","Data":"d1e4d51018f071df87f92ad49e614b254c2a429ad49767b2956c9752e532ea36"}
May 25 12:12:49.115874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:49.115811    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-91.ec2.internal" podStartSLOduration=4.11578856 podStartE2EDuration="4.11578856s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:12:48.073961256 +0000 UTC m=+3.732292662" watchObservedRunningTime="2026-05-25 12:12:49.11578856 +0000 UTC m=+4.774119968"
May 25 12:12:49.459171 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:49.459134    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:49.459367 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:49.459300    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:12:49.459431 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:49.459365    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret podName:3f71c2f0-743b-4295-af1d-2182812eaeb6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:51.459346431 +0000 UTC m=+7.117677820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret") pod "global-pull-secret-syncer-wlsj7" (UID: "3f71c2f0-743b-4295-af1d-2182812eaeb6") : object "kube-system"/"original-pull-secret" not registered
May 25 12:12:49.950962 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:49.950381    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:49.950962 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:49.950561    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:12:50.467900 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:50.467860    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:50.468345 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:50.468026    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:50.468345 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:50.468094    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs podName:037bfb47-48b6-4248-9537-b2da35bdf80f nodeName:}" failed. No retries permitted until 2026-05-25 12:12:54.46807823 +0000 UTC m=+10.126409617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs") pod "network-metrics-daemon-22jr7" (UID: "037bfb47-48b6-4248-9537-b2da35bdf80f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:50.568994 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:50.568947    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:50.569203 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:50.569186    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:50.569286 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:50.569210    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:50.569286 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:50.569224    2562 projected.go:194] Error preparing data for projected volume kube-api-access-h2dj7 for pod openshift-network-diagnostics/network-check-target-rvhv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:50.569398 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:50.569300    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7 podName:0a521353-c135-45c9-a6f1-0d36c801505d nodeName:}" failed. No retries permitted until 2026-05-25 12:12:54.569279173 +0000 UTC m=+10.227610560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2dj7" (UniqueName: "kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7") pod "network-check-target-rvhv8" (UID: "0a521353-c135-45c9-a6f1-0d36c801505d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:50.951111 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:50.951078    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:50.951302 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:50.951230    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:12:50.951395 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:50.951319    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:50.951454 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:50.951416    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:12:51.476679 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:51.476487    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:51.476679 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:51.476636    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:12:51.477207 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:51.476704    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret podName:3f71c2f0-743b-4295-af1d-2182812eaeb6 nodeName:}" failed. No retries permitted until 2026-05-25 12:12:55.476685363 +0000 UTC m=+11.135016746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret") pod "global-pull-secret-syncer-wlsj7" (UID: "3f71c2f0-743b-4295-af1d-2182812eaeb6") : object "kube-system"/"original-pull-secret" not registered
May 25 12:12:51.950633 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:51.950532    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:51.950799 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:51.950672    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:12:52.950868 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:52.950751    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:52.951349 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:52.950883    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:12:52.951423 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:52.951397    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:52.951564 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:52.951520    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:12:53.950800 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:53.950757    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:53.950974 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:53.950904    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:12:54.500983 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:54.500936    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:54.501210 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:54.501113    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:54.501210 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:54.501190    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs podName:037bfb47-48b6-4248-9537-b2da35bdf80f nodeName:}" failed. No retries permitted until 2026-05-25 12:13:02.501171356 +0000 UTC m=+18.159502753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs") pod "network-metrics-daemon-22jr7" (UID: "037bfb47-48b6-4248-9537-b2da35bdf80f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:54.602175 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:54.602134    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:54.602398 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:54.602370    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:54.602398 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:54.602397    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:54.602516 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:54.602411    2562 projected.go:194] Error preparing data for projected volume kube-api-access-h2dj7 for pod openshift-network-diagnostics/network-check-target-rvhv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:54.602516 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:54.602484    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7 podName:0a521353-c135-45c9-a6f1-0d36c801505d nodeName:}" failed. No retries permitted until 2026-05-25 12:13:02.60245965 +0000 UTC m=+18.260791047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2dj7" (UniqueName: "kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7") pod "network-check-target-rvhv8" (UID: "0a521353-c135-45c9-a6f1-0d36c801505d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:54.951568 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:54.951473    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:54.952022 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:54.951597    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:54.952022 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:54.951696    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:12:54.952022 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:54.951757    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:12:55.508818 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:55.508778    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:55.508999 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:55.508921    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:12:55.508999 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:55.508988    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret podName:3f71c2f0-743b-4295-af1d-2182812eaeb6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:03.508970151 +0000 UTC m=+19.167301536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret") pod "global-pull-secret-syncer-wlsj7" (UID: "3f71c2f0-743b-4295-af1d-2182812eaeb6") : object "kube-system"/"original-pull-secret" not registered
May 25 12:12:55.950232 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:55.950146    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:55.950423 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:55.950296    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:12:56.950202 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:56.950159    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:56.950637 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:56.950175    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:56.950637 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:56.950334    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:12:56.950637 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:56.950418    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:12:57.951108 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:57.950986    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:57.951570 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:57.951122    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:12:58.950875 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:58.950841    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:12:58.951040 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:58.950980    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:12:58.951094 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:58.951044    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:12:58.951198 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:58.951179    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:12:59.951078 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:12:59.951044    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:12:59.951240 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:12:59.951146    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:00.950926 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:00.950882    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:00.951086 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:00.951031    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:00.951142 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:00.951117    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:00.951273 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:00.951236    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:01.950408 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:01.950371    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:01.950627 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:01.950495    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:02.562170 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:02.562136    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:02.562658 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:02.562323    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:02.562658 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:02.562409    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs podName:037bfb47-48b6-4248-9537-b2da35bdf80f nodeName:}" failed. No retries permitted until 2026-05-25 12:13:18.562386557 +0000 UTC m=+34.220717940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs") pod "network-metrics-daemon-22jr7" (UID: "037bfb47-48b6-4248-9537-b2da35bdf80f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:02.663020 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:02.662971    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:02.663206 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:02.663154    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:02.663206 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:02.663182    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:02.663206 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:02.663196    2562 projected.go:194] Error preparing data for projected volume kube-api-access-h2dj7 for pod openshift-network-diagnostics/network-check-target-rvhv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:02.663384 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:02.663279    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7 podName:0a521353-c135-45c9-a6f1-0d36c801505d nodeName:}" failed. No retries permitted until 2026-05-25 12:13:18.663242335 +0000 UTC m=+34.321573722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2dj7" (UniqueName: "kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7") pod "network-check-target-rvhv8" (UID: "0a521353-c135-45c9-a6f1-0d36c801505d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:02.951160 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:02.951084    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:02.951338 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:02.951087    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:02.951338 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:02.951217    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:02.951338 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:02.951311    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:03.570233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:03.570189    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:03.570640 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:03.570378    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:13:03.570640 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:03.570457    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret podName:3f71c2f0-743b-4295-af1d-2182812eaeb6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.570437523 +0000 UTC m=+35.228768919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret") pod "global-pull-secret-syncer-wlsj7" (UID: "3f71c2f0-743b-4295-af1d-2182812eaeb6") : object "kube-system"/"original-pull-secret" not registered
May 25 12:13:03.950444 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:03.950370    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:03.950595 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:03.950497    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:04.951304 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:04.951053    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:04.951733 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:04.951125    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:04.951733 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:04.951382    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:04.951733 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:04.951530    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:05.950683 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:05.950657    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:05.950839 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:05.950747    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:06.951304 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:06.951070    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:06.951304 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:06.951113    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:06.951961 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:06.951354    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:06.951961 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:06.951457    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:07.140825 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.140732    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7hkxn" event={"ID":"994d3f70-8d53-4232-9eec-f97966e0fa11","Type":"ContainerStarted","Data":"a1bcf7390bcc6744c6a74ba9768911939e37d65aaeb1445801ed797e311b5207"}
May 25 12:13:07.143354 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.143323    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" event={"ID":"96d029b0-6c0f-496a-b1f0-5a651825fc29","Type":"ContainerStarted","Data":"0a22b42296d6afff6f723bc9d6bdfe4a5dbcd1b9ad8b922d411f9c861fef556d"}
May 25 12:13:07.143354 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.143354    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" event={"ID":"96d029b0-6c0f-496a-b1f0-5a651825fc29","Type":"ContainerStarted","Data":"668ce7c453f54c94ff7c9af22fc295ad744afa2102c3c7fa2c742382cbe22e31"}
May 25 12:13:07.143532 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.143368    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" event={"ID":"96d029b0-6c0f-496a-b1f0-5a651825fc29","Type":"ContainerStarted","Data":"1142ba79bd2e362b8d7d5855c6105ba007c5c93bb312b5a32a0d06bca96544e2"}
May 25 12:13:07.143532 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.143393    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" event={"ID":"96d029b0-6c0f-496a-b1f0-5a651825fc29","Type":"ContainerStarted","Data":"985910db785a10faec8b08522140a4b18a149e1b7bf2dc001e5ee275f57d6fa7"}
May 25 12:13:07.143532 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.143402    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" event={"ID":"96d029b0-6c0f-496a-b1f0-5a651825fc29","Type":"ContainerStarted","Data":"d0d45cf540ba2d85ae5bcae0015f98da4e9f449dc2d3de5e6907cffde476dc70"}
May 25 12:13:07.143532 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.143410    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" event={"ID":"96d029b0-6c0f-496a-b1f0-5a651825fc29","Type":"ContainerStarted","Data":"63a4453528cb2d58491c4cd968c905832167f6288d1cc8818f8d1f04bb09c428"}
May 25 12:13:07.144608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.144588    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9q65" event={"ID":"01253226-6d29-4eac-9a58-3d9a8be0032d","Type":"ContainerStarted","Data":"7916b0be568788fec62b9261c4750b91626d5b1bbd373001a569dbeb43ab3c93"}
May 25 12:13:07.145879 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.145862    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cf9x4" event={"ID":"cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f","Type":"ContainerStarted","Data":"5d3b4461ceda97fe35ea20a1576af261d6e13103ea95c1426145fa16a49f6f33"}
May 25 12:13:07.147142 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.147119    2562 generic.go:358] "Generic (PLEG): container finished" podID="e1367fb0-fb43-4995-9af7-9be8baf478af" containerID="34e0a208b1247a7e327a9d4d2aa820fe1e7daffabd4a02e5885f6cd72d32580e" exitCode=0
May 25 12:13:07.147228 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.147186    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sfdq" event={"ID":"e1367fb0-fb43-4995-9af7-9be8baf478af","Type":"ContainerDied","Data":"34e0a208b1247a7e327a9d4d2aa820fe1e7daffabd4a02e5885f6cd72d32580e"}
May 25 12:13:07.148446 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.148422    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dg8fc" event={"ID":"128d93e5-705b-4d8f-b4b0-60af82c40726","Type":"ContainerStarted","Data":"d54d23366c556104604216865f61493ccf584f09384d785a81cc0271755a1746"}
May 25 12:13:07.149703 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.149683    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tmtnm" event={"ID":"249e00b6-ef97-49ed-8be8-069779eace7d","Type":"ContainerStarted","Data":"a8221447d5c717834652e169ab30840e72f22ffa9dba8c79179e9564e334f5ea"}
May 25 12:13:07.150908 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.150889    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs" event={"ID":"77f68379-dc11-4aaf-9179-daf22dfddab1","Type":"ContainerStarted","Data":"5866bd3e89e7f5fbd9805c8ee12d3838c8330b980d8675171463f1626746b7be"}
May 25 12:13:07.169186 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.169104    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-91.ec2.internal" podStartSLOduration=22.169090056 podStartE2EDuration="22.169090056s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:12:49.116697337 +0000 UTC m=+4.775028743" watchObservedRunningTime="2026-05-25 12:13:07.169090056 +0000 UTC m=+22.827421463"
May 25 12:13:07.169388 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.169215    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-7hkxn" podStartSLOduration=3.624101675 podStartE2EDuration="22.169206608s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="2026-05-25 12:12:47.46647474 +0000 UTC m=+3.124806129" lastFinishedPulling="2026-05-25 12:13:06.011579474 +0000 UTC m=+21.669911062" observedRunningTime="2026-05-25 12:13:07.168608443 +0000 UTC m=+22.826939850" watchObservedRunningTime="2026-05-25 12:13:07.169206608 +0000 UTC m=+22.827538012"
May 25 12:13:07.198345 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.198299    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tmtnm" podStartSLOduration=3.664721999 podStartE2EDuration="22.198285168s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="2026-05-25 12:12:47.470648406 +0000 UTC m=+3.128979800" lastFinishedPulling="2026-05-25 12:13:06.00421157 +0000 UTC m=+21.662542969" observedRunningTime="2026-05-25 12:13:07.183369923 +0000 UTC m=+22.841701328" watchObservedRunningTime="2026-05-25 12:13:07.198285168 +0000 UTC m=+22.856616572"
May 25 12:13:07.198516 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.198394    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cf9x4" podStartSLOduration=3.667416931 podStartE2EDuration="22.198390744s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="2026-05-25 12:12:47.471508851 +0000 UTC m=+3.129840238" lastFinishedPulling="2026-05-25 12:13:06.002482662 +0000 UTC m=+21.660814051" observedRunningTime="2026-05-25 12:13:07.197994695 +0000 UTC m=+22.856326100" watchObservedRunningTime="2026-05-25 12:13:07.198390744 +0000 UTC m=+22.856722153"
May 25 12:13:07.244659 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.244617    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dg8fc" podStartSLOduration=13.026554483 podStartE2EDuration="23.244601921s" podCreationTimestamp="2026-05-25 12:12:44 +0000 UTC" firstStartedPulling="2026-05-25 12:12:47.472390707 +0000 UTC m=+3.130722103" lastFinishedPulling="2026-05-25 12:12:57.690438158 +0000 UTC m=+13.348769541" observedRunningTime="2026-05-25 12:13:07.244327488 +0000 UTC m=+22.902658885" watchObservedRunningTime="2026-05-25 12:13:07.244601921 +0000 UTC m=+22.902933325"
May 25 12:13:07.304731 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.304681    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-j9q65" podStartSLOduration=3.761998736 podStartE2EDuration="22.304667408s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="2026-05-25 12:12:47.462057378 +0000 UTC m=+3.120388779" lastFinishedPulling="2026-05-25 12:13:06.004726063 +0000 UTC m=+21.663057451" observedRunningTime="2026-05-25 12:13:07.304312648 +0000 UTC m=+22.962644054" watchObservedRunningTime="2026-05-25 12:13:07.304667408 +0000 UTC m=+22.962998813"
May 25 12:13:07.334525 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.334500    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 12:13:07.889343 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.889219    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T12:13:07.334521448Z","UUID":"31ac06cc-2a5a-4c4a-956f-76467cd14543","Handler":null,"Name":"","Endpoint":""}
May 25 12:13:07.891395 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.891368    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 12:13:07.891395 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.891397    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 12:13:07.950687 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:07.950656    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:07.950864 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:07.950776    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:08.154303 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:08.154144    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2d962" event={"ID":"782e7027-de68-4d9f-83de-04b38965cd87","Type":"ContainerStarted","Data":"f6f0b76da278fd7dad1ff555f1573f0ca6874df862db7319d40a632b9d0c0c6f"}
May 25 12:13:08.156604 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:08.156573    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs" event={"ID":"77f68379-dc11-4aaf-9179-daf22dfddab1","Type":"ContainerStarted","Data":"dc49c3747962128b4edc11e69fee350540799975b98b43d8e6b5e47a0f9a9f83"}
May 25 12:13:08.950513 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:08.950363    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:08.950513 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:08.950388    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:08.950743 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:08.950594    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:08.950743 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:08.950653    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:09.161173 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:09.161131    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" event={"ID":"96d029b0-6c0f-496a-b1f0-5a651825fc29","Type":"ContainerStarted","Data":"5f561ed7b4d1ab699a45d823ea4f8df96d0f7c646df431b82202a8c8d83b38c3"}
May 25 12:13:09.163071 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:09.163042    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs" event={"ID":"77f68379-dc11-4aaf-9179-daf22dfddab1","Type":"ContainerStarted","Data":"ce5446a5b35a7ee95bc7f1458cd7fdb606d075291e0dbd3e815cbd932b272dd8"}
May 25 12:13:09.179474 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:09.179390    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2d962" podStartSLOduration=5.650165254 podStartE2EDuration="24.179374594s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="2026-05-25 12:12:47.473350748 +0000 UTC m=+3.131682134" lastFinishedPulling="2026-05-25 12:13:06.002560076 +0000 UTC m=+21.660891474" observedRunningTime="2026-05-25 12:13:08.167770544 +0000 UTC m=+23.826101949" watchObservedRunningTime="2026-05-25 12:13:09.179374594 +0000 UTC m=+24.837705999"
May 25 12:13:09.180018 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:09.179981    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kgdfs" podStartSLOduration=3.465953985 podStartE2EDuration="24.179968736s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="2026-05-25 12:12:47.470419075 +0000 UTC m=+3.128750458" lastFinishedPulling="2026-05-25 12:13:08.184433809 +0000 UTC m=+23.842765209" observedRunningTime="2026-05-25 12:13:09.179115621 +0000 UTC m=+24.837447026" watchObservedRunningTime="2026-05-25 12:13:09.179968736 +0000 UTC m=+24.838300144"
May 25 12:13:09.951117 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:09.951079    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:09.951323 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:09.951211    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:10.950687 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:10.950644    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:10.951311 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:10.950690    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:10.951311 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:10.950802    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:10.951311 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:10.950914    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:11.950693 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:11.950515    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:11.951141 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:11.950763    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:12.030937 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.030902    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:13:12.031948 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.031928    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:13:12.171423 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.171386    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" event={"ID":"96d029b0-6c0f-496a-b1f0-5a651825fc29","Type":"ContainerStarted","Data":"76f29d5bd2b7eb8a9ddb3529bc0d8aa849680f78426394ecca37017bfaf525f2"}
May 25 12:13:12.171691 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.171669    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:13:12.171795 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.171700    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:13:12.173065 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.173042    2562 generic.go:358] "Generic (PLEG): container finished" podID="e1367fb0-fb43-4995-9af7-9be8baf478af" containerID="a5d73ba57f6b2a164e5f6075352fee2003ca43f6e59f8d7d8065ff46e6aefb40" exitCode=0
May 25 12:13:12.173170 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.173131    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sfdq" event={"ID":"e1367fb0-fb43-4995-9af7-9be8baf478af","Type":"ContainerDied","Data":"a5d73ba57f6b2a164e5f6075352fee2003ca43f6e59f8d7d8065ff46e6aefb40"}
May 25 12:13:12.187473 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.187453    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:13:12.197435 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.197400    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl" podStartSLOduration=8.385161949 podStartE2EDuration="27.197387482s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="2026-05-25 12:12:47.466008338 +0000 UTC m=+3.124339726" lastFinishedPulling="2026-05-25 12:13:06.278233871 +0000 UTC m=+21.936565259" observedRunningTime="2026-05-25 12:13:12.19605976 +0000 UTC m=+27.854391165" watchObservedRunningTime="2026-05-25 12:13:12.197387482 +0000 UTC m=+27.855718868"
May 25 12:13:12.950483 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.950452    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:12.950634 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:12.950452    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:12.950696 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:12.950543    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:12.950696 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:12.950673    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:13.130081 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.130043    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rvhv8"]
May 25 12:13:13.130217 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.130187    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:13.130351 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:13.130330    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:13.132990 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.132961    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wlsj7"]
May 25 12:13:13.133610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.133560    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-22jr7"]
May 25 12:13:13.177018 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.176990    2562 generic.go:358] "Generic (PLEG): container finished" podID="e1367fb0-fb43-4995-9af7-9be8baf478af" containerID="b51444e51facf24bab715ad7006a04f6e79279f72ea20cc7e4c05e53cc3c9915" exitCode=0
May 25 12:13:13.177168 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.177110    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sfdq" event={"ID":"e1367fb0-fb43-4995-9af7-9be8baf478af","Type":"ContainerDied","Data":"b51444e51facf24bab715ad7006a04f6e79279f72ea20cc7e4c05e53cc3c9915"}
May 25 12:13:13.177228 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.177211    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:13.177339 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.177321    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:13.177446 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:13.177427    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:13.177446 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:13.177328    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:13.178064 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.177755    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:13:13.192666 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:13.192647    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:13:14.181178 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:14.180918    2562 generic.go:358] "Generic (PLEG): container finished" podID="e1367fb0-fb43-4995-9af7-9be8baf478af" containerID="62095d5f05ac10b1e7e6185426028c1cd28a9eaa20b5274ab47e5f819341338f" exitCode=0
May 25 12:13:14.181178 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:14.181004    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sfdq" event={"ID":"e1367fb0-fb43-4995-9af7-9be8baf478af","Type":"ContainerDied","Data":"62095d5f05ac10b1e7e6185426028c1cd28a9eaa20b5274ab47e5f819341338f"}
May 25 12:13:14.511973 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:14.511943    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:13:14.512123 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:14.512075    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 25 12:13:14.512544 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:14.512529    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dg8fc"
May 25 12:13:14.951206 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:14.951125    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:14.951363 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:14.951249    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:14.951363 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:14.951343    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:14.951495 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:14.951463    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:14.951623 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:14.951521    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:14.951623 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:14.951614    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:16.950806 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:16.950769    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:16.951559 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:16.950897    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:16.951559 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:16.950911    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:16.951559 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:16.951008    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:16.951559 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:16.951057    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:16.951559 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:16.951132    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:18.579056 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:18.579019    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:18.579590 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:18.579197    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:18.579590 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:18.579294    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs podName:037bfb47-48b6-4248-9537-b2da35bdf80f nodeName:}" failed. No retries permitted until 2026-05-25 12:13:50.5792726 +0000 UTC m=+66.237604003 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs") pod "network-metrics-daemon-22jr7" (UID: "037bfb47-48b6-4248-9537-b2da35bdf80f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:18.679579 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:18.679547    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:18.679747 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:18.679705    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:18.679747 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:18.679739    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:18.679747 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:18.679751    2562 projected.go:194] Error preparing data for projected volume kube-api-access-h2dj7 for pod openshift-network-diagnostics/network-check-target-rvhv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:18.679872 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:18.679812    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7 podName:0a521353-c135-45c9-a6f1-0d36c801505d nodeName:}" failed. No retries permitted until 2026-05-25 12:13:50.679796351 +0000 UTC m=+66.338127737 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2dj7" (UniqueName: "kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7") pod "network-check-target-rvhv8" (UID: "0a521353-c135-45c9-a6f1-0d36c801505d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:18.951003 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:18.950848    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:18.951003 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:18.950848    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:18.951225 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:18.950853    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:18.951225 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:18.950995    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:13:18.951225 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:18.951073    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvhv8" podUID="0a521353-c135-45c9-a6f1-0d36c801505d"
May 25 12:13:18.951225 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:18.951168    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wlsj7" podUID="3f71c2f0-743b-4295-af1d-2182812eaeb6"
May 25 12:13:19.147450 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.147415    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-91.ec2.internal" event="NodeReady"
May 25 12:13:19.147629 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.147569    2562 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 12:13:19.178851 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.178818    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"]
May 25 12:13:19.209644 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.209558    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"]
May 25 12:13:19.210284 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.210148    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.213239 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.213215    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 25 12:13:19.213382 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.213241    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 25 12:13:19.213550 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.213498    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 25 12:13:19.213550 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.213548    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 25 12:13:19.225012 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.224986    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"]
May 25 12:13:19.225130 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.225117    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.228013 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.227994    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 12:13:19.228013 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.228003    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 12:13:19.228193 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.228022    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k9dhz\""
May 25 12:13:19.228193 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.228133    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 12:13:19.233765 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.233743    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 12:13:19.241128 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.241104    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"]
May 25 12:13:19.241327 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.241309    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:19.244055 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.244032    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 25 12:13:19.244152 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.244086    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 25 12:13:19.244152 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.244094    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-22tjv\""
May 25 12:13:19.259689 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.259668    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"]
May 25 12:13:19.259853 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.259836    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"
May 25 12:13:19.262677 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.262656    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-gkc5v\""
May 25 12:13:19.262785 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.262756    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 25 12:13:19.277894 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.277873    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"]
May 25 12:13:19.278007 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.277901    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"]
May 25 12:13:19.278007 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.277913    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"]
May 25 12:13:19.278007 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.277928    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bc7wq"]
May 25 12:13:19.278150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.278019    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.281049 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.280828    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 25 12:13:19.281049 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.280862    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 25 12:13:19.281049 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.280872    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 25 12:13:19.281049 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.280834    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 25 12:13:19.282894 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.282871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4k9z\" (UniqueName: \"kubernetes.io/projected/2097934f-3b96-486a-aa25-59074840b23e-kube-api-access-b4k9z\") pod \"klusterlet-addon-workmgr-5c87bf6ff-24zbr\" (UID: \"2097934f-3b96-486a-aa25-59074840b23e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.282995 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.282915    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2097934f-3b96-486a-aa25-59074840b23e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c87bf6ff-24zbr\" (UID: \"2097934f-3b96-486a-aa25-59074840b23e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.283094 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.283013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2097934f-3b96-486a-aa25-59074840b23e-tmp\") pod \"klusterlet-addon-workmgr-5c87bf6ff-24zbr\" (UID: \"2097934f-3b96-486a-aa25-59074840b23e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.301987 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.301952    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"]
May 25 12:13:19.301987 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.301990    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"]
May 25 12:13:19.302190 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.302000    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bc7wq"]
May 25 12:13:19.302190 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.302025    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c77cg"]
May 25 12:13:19.302190 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.301999    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:19.305225 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.305193    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 12:13:19.305359 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.305234    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zkb72\""
May 25 12:13:19.305359 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.305331    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 12:13:19.305481 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.305359    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 12:13:19.321227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.321202    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c77cg"]
May 25 12:13:19.321362 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.321342    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.323961 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.323940    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 12:13:19.323961 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.323940    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 12:13:19.324191 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.323940    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fp9z9\""
May 25 12:13:19.384222 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-registry-certificates\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.384422 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-image-registry-private-configuration\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.384422 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt82f\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-kube-api-access-dt82f\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.384422 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1ed9528-15dc-44a5-94ce-b457a0eccece-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:19.384422 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384412    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-ca\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.384665 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384453    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-bound-sa-token\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.384665 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:19.384665 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384538    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a94f30d8-539a-4618-9075-95c2295727d8-ca-trust-extracted\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.384665 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:19.384665 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lwm5\" (UniqueName: \"kubernetes.io/projected/06c76375-02e4-4984-a004-7e997033a7f9-kube-api-access-8lwm5\") pod \"managed-serviceaccount-addon-agent-85b9c949c6-cqn2v\" (UID: \"06c76375-02e4-4984-a004-7e997033a7f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"
May 25 12:13:19.384665 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-installation-pull-secrets\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.384665 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-hub\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.384916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.384916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.384916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384730    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2097934f-3b96-486a-aa25-59074840b23e-tmp\") pod \"klusterlet-addon-workmgr-5c87bf6ff-24zbr\" (UID: \"2097934f-3b96-486a-aa25-59074840b23e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.384916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384759    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.384916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384787    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7m2r\" (UniqueName: \"kubernetes.io/projected/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-kube-api-access-k7m2r\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.384916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8g5g\" (UniqueName: \"kubernetes.io/projected/6ebba530-a3c2-4deb-918c-b0f9ca60c101-kube-api-access-l8g5g\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:19.384916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384903    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b4k9z\" (UniqueName: \"kubernetes.io/projected/2097934f-3b96-486a-aa25-59074840b23e-kube-api-access-b4k9z\") pod \"klusterlet-addon-workmgr-5c87bf6ff-24zbr\" (UID: \"2097934f-3b96-486a-aa25-59074840b23e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.385153 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-trusted-ca\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.385153 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384962    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2097934f-3b96-486a-aa25-59074840b23e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c87bf6ff-24zbr\" (UID: \"2097934f-3b96-486a-aa25-59074840b23e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.385153 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.384989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06c76375-02e4-4984-a004-7e997033a7f9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85b9c949c6-cqn2v\" (UID: \"06c76375-02e4-4984-a004-7e997033a7f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"
May 25 12:13:19.385153 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.385044    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.385153 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.385144    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2097934f-3b96-486a-aa25-59074840b23e-tmp\") pod \"klusterlet-addon-workmgr-5c87bf6ff-24zbr\" (UID: \"2097934f-3b96-486a-aa25-59074840b23e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.389676 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.389651    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2097934f-3b96-486a-aa25-59074840b23e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c87bf6ff-24zbr\" (UID: \"2097934f-3b96-486a-aa25-59074840b23e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.393711 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.393662    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4k9z\" (UniqueName: \"kubernetes.io/projected/2097934f-3b96-486a-aa25-59074840b23e-kube-api-access-b4k9z\") pod \"klusterlet-addon-workmgr-5c87bf6ff-24zbr\" (UID: \"2097934f-3b96-486a-aa25-59074840b23e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.485635 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.485602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffcm\" (UniqueName: \"kubernetes.io/projected/059085b8-4425-4d8c-b229-bd2bce5c1657-kube-api-access-nffcm\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.485812 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.485654    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.485812 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.485698    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-trusted-ca\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.485929 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.485848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059085b8-4425-4d8c-b229-bd2bce5c1657-tmp-dir\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.485929 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.485915    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.486016 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.485942    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-registry-certificates\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.486016 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.485967    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-hub\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.486016 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.485998    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:19.486165 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486021    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dt82f\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-kube-api-access-dt82f\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.486165 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486037    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-ca\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.486165 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486053    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l8g5g\" (UniqueName: \"kubernetes.io/projected/6ebba530-a3c2-4deb-918c-b0f9ca60c101-kube-api-access-l8g5g\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:19.486165 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486091    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:19.486165 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486117    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a94f30d8-539a-4618-9075-95c2295727d8-ca-trust-extracted\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.486165 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486149    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-installation-pull-secrets\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.486468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486175    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.486468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486200    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k7m2r\" (UniqueName: \"kubernetes.io/projected/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-kube-api-access-k7m2r\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.486468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486241    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06c76375-02e4-4984-a004-7e997033a7f9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85b9c949c6-cqn2v\" (UID: \"06c76375-02e4-4984-a004-7e997033a7f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"
May 25 12:13:19.486468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/059085b8-4425-4d8c-b229-bd2bce5c1657-config-volume\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.486468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486347    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.486468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486373    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-bound-sa-token\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.486468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486410    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-image-registry-private-configuration\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.486468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486440    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8lwm5\" (UniqueName: \"kubernetes.io/projected/06c76375-02e4-4984-a004-7e997033a7f9-kube-api-access-8lwm5\") pod \"managed-serviceaccount-addon-agent-85b9c949c6-cqn2v\" (UID: \"06c76375-02e4-4984-a004-7e997033a7f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"
May 25 12:13:19.486468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.486891 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486514    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1ed9528-15dc-44a5-94ce-b457a0eccece-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:19.486891 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486714    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-trusted-ca\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.486891 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.486735    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.487154 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.487132    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 12:13:19.487290 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.487191    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert podName:a1ed9528-15dc-44a5-94ce-b457a0eccece nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.987174801 +0000 UTC m=+35.645506185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-httbc" (UID: "a1ed9528-15dc-44a5-94ce-b457a0eccece") : secret "networking-console-plugin-cert" not found
May 25 12:13:19.487290 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.487228    2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:19.487290 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.487236    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1ed9528-15dc-44a5-94ce-b457a0eccece-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:19.487290 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.487248    2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c68f5fcb8-qv8f5: secret "image-registry-tls" not found
May 25 12:13:19.487527 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.487314    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls podName:a94f30d8-539a-4618-9075-95c2295727d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.987296394 +0000 UTC m=+35.645627789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls") pod "image-registry-5c68f5fcb8-qv8f5" (UID: "a94f30d8-539a-4618-9075-95c2295727d8") : secret "image-registry-tls" not found
May 25 12:13:19.487811 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.487780    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-registry-certificates\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.488465 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.487983    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:19.488465 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.488036    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert podName:6ebba530-a3c2-4deb-918c-b0f9ca60c101 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.988021085 +0000 UTC m=+35.646352473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert") pod "ingress-canary-bc7wq" (UID: "6ebba530-a3c2-4deb-918c-b0f9ca60c101") : secret "canary-serving-cert" not found
May 25 12:13:19.488465 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.488348    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a94f30d8-539a-4618-9075-95c2295727d8-ca-trust-extracted\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.490094 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.490051    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-installation-pull-secrets\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.490208 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.490179    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.490332 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.490229    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-ca\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.490928 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.490882    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06c76375-02e4-4984-a004-7e997033a7f9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-85b9c949c6-cqn2v\" (UID: \"06c76375-02e4-4984-a004-7e997033a7f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"
May 25 12:13:19.491244 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.491224    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-image-registry-private-configuration\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.491405 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.491380    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-hub\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.491535 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.491514    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.499001 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.498976    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8g5g\" (UniqueName: \"kubernetes.io/projected/6ebba530-a3c2-4deb-918c-b0f9ca60c101-kube-api-access-l8g5g\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:19.499123 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.499080    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7m2r\" (UniqueName: \"kubernetes.io/projected/d7ea3aea-da4a-49d5-be77-4de8dddf91bb-kube-api-access-k7m2r\") pod \"cluster-proxy-proxy-agent-cddc9cdc-jkdgr\" (UID: \"d7ea3aea-da4a-49d5-be77-4de8dddf91bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.499231 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.499207    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt82f\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-kube-api-access-dt82f\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.499354 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.499285    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-bound-sa-token\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.500325 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.500302    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lwm5\" (UniqueName: \"kubernetes.io/projected/06c76375-02e4-4984-a004-7e997033a7f9-kube-api-access-8lwm5\") pod \"managed-serviceaccount-addon-agent-85b9c949c6-cqn2v\" (UID: \"06c76375-02e4-4984-a004-7e997033a7f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"
May 25 12:13:19.522797 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.522760    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:19.581659 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.581608    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"
May 25 12:13:19.587626 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.587603    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/059085b8-4425-4d8c-b229-bd2bce5c1657-config-volume\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.587752 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.587648    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.587811 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.587753    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:19.587811 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.587795    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nffcm\" (UniqueName: \"kubernetes.io/projected/059085b8-4425-4d8c-b229-bd2bce5c1657-kube-api-access-nffcm\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.587912 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.587818    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls podName:059085b8-4425-4d8c-b229-bd2bce5c1657 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:20.087799815 +0000 UTC m=+35.746131197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls") pod "dns-default-c77cg" (UID: "059085b8-4425-4d8c-b229-bd2bce5c1657") : secret "dns-default-metrics-tls" not found
May 25 12:13:19.587912 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.587856    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:19.587912 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.587883    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059085b8-4425-4d8c-b229-bd2bce5c1657-tmp-dir\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.588067 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.587969    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:13:19.588067 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.588031    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret podName:3f71c2f0-743b-4295-af1d-2182812eaeb6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:51.588014068 +0000 UTC m=+67.246345471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret") pod "global-pull-secret-syncer-wlsj7" (UID: "3f71c2f0-743b-4295-af1d-2182812eaeb6") : object "kube-system"/"original-pull-secret" not registered
May 25 12:13:19.588200 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.588182    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059085b8-4425-4d8c-b229-bd2bce5c1657-tmp-dir\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.588305 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.588288    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/059085b8-4425-4d8c-b229-bd2bce5c1657-config-volume\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.589577 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.589559    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:13:19.605574 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.605549    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffcm\" (UniqueName: \"kubernetes.io/projected/059085b8-4425-4d8c-b229-bd2bce5c1657-kube-api-access-nffcm\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:19.991047 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.991003    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:19.991252 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.991068    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:19.991252 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:19.991117    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:19.991252 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.991159    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 12:13:19.991252 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.991228    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert podName:a1ed9528-15dc-44a5-94ce-b457a0eccece nodeName:}" failed. No retries permitted until 2026-05-25 12:13:20.991208724 +0000 UTC m=+36.649540124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-httbc" (UID: "a1ed9528-15dc-44a5-94ce-b457a0eccece") : secret "networking-console-plugin-cert" not found
May 25 12:13:19.991252 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.991233    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:19.991544 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.991312    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert podName:6ebba530-a3c2-4deb-918c-b0f9ca60c101 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:20.991294094 +0000 UTC m=+36.649625491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert") pod "ingress-canary-bc7wq" (UID: "6ebba530-a3c2-4deb-918c-b0f9ca60c101") : secret "canary-serving-cert" not found
May 25 12:13:19.991544 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.991244    2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:19.991544 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.991336    2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c68f5fcb8-qv8f5: secret "image-registry-tls" not found
May 25 12:13:19.991544 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:19.991377    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls podName:a94f30d8-539a-4618-9075-95c2295727d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:20.991366019 +0000 UTC m=+36.649697406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls") pod "image-registry-5c68f5fcb8-qv8f5" (UID: "a94f30d8-539a-4618-9075-95c2295727d8") : secret "image-registry-tls" not found
May 25 12:13:20.092069 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.092033    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:20.092281 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:20.092178    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:20.092281 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:20.092272    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls podName:059085b8-4425-4d8c-b229-bd2bce5c1657 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:21.092236437 +0000 UTC m=+36.750567821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls") pod "dns-default-c77cg" (UID: "059085b8-4425-4d8c-b229-bd2bce5c1657") : secret "dns-default-metrics-tls" not found
May 25 12:13:20.386070 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.385860    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v"]
May 25 12:13:20.392783 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.392761    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"]
May 25 12:13:20.398756 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.398147    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"]
May 25 12:13:20.519836 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:13:20.519779    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2097934f_3b96_486a_aa25_59074840b23e.slice/crio-b431f7192b148d8f95cdf8db81d53dd6b603fd550e9daa3fc1d43c5203d47b09 WatchSource:0}: Error finding container b431f7192b148d8f95cdf8db81d53dd6b603fd550e9daa3fc1d43c5203d47b09: Status 404 returned error can't find the container with id b431f7192b148d8f95cdf8db81d53dd6b603fd550e9daa3fc1d43c5203d47b09
May 25 12:13:20.520503 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:13:20.520476    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7ea3aea_da4a_49d5_be77_4de8dddf91bb.slice/crio-ca1e84a2c78359941cc8ec8f1617754fb1e51ef1c403568d066b00dabc6f42cc WatchSource:0}: Error finding container ca1e84a2c78359941cc8ec8f1617754fb1e51ef1c403568d066b00dabc6f42cc: Status 404 returned error can't find the container with id ca1e84a2c78359941cc8ec8f1617754fb1e51ef1c403568d066b00dabc6f42cc
May 25 12:13:20.950637 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.950601    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:20.950637 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.950623    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:20.951200 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.950702    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:20.955314 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.955292    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:13:20.955472 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.955314    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:13:20.955472 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.955321    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxdt4\""
May 25 12:13:20.955472 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.955328    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:13:20.955619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.955607    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nlm6z\""
May 25 12:13:20.965827 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.965807    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 12:13:20.997781 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.997741    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:20.997935 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.997790    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:20.997935 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:20.997900    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:20.997935 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:20.997905    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 12:13:20.998104 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:20.997967    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert podName:6ebba530-a3c2-4deb-918c-b0f9ca60c101 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:22.997948956 +0000 UTC m=+38.656280340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert") pod "ingress-canary-bc7wq" (UID: "6ebba530-a3c2-4deb-918c-b0f9ca60c101") : secret "canary-serving-cert" not found
May 25 12:13:20.998104 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:20.997988    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert podName:a1ed9528-15dc-44a5-94ce-b457a0eccece nodeName:}" failed. No retries permitted until 2026-05-25 12:13:22.997978203 +0000 UTC m=+38.656309587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-httbc" (UID: "a1ed9528-15dc-44a5-94ce-b457a0eccece") : secret "networking-console-plugin-cert" not found
May 25 12:13:20.998104 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:20.998021    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:20.998234 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:20.998148    2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:20.998234 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:20.998159    2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c68f5fcb8-qv8f5: secret "image-registry-tls" not found
May 25 12:13:20.998234 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:20.998197    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls podName:a94f30d8-539a-4618-9075-95c2295727d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:22.998184014 +0000 UTC m=+38.656515418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls") pod "image-registry-5c68f5fcb8-qv8f5" (UID: "a94f30d8-539a-4618-9075-95c2295727d8") : secret "image-registry-tls" not found
May 25 12:13:21.099482 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:21.099398    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:21.099645 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:21.099538    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:21.099645 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:21.099604    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls podName:059085b8-4425-4d8c-b229-bd2bce5c1657 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:23.099583832 +0000 UTC m=+38.757915232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls") pod "dns-default-c77cg" (UID: "059085b8-4425-4d8c-b229-bd2bce5c1657") : secret "dns-default-metrics-tls" not found
May 25 12:13:21.198968 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:21.198932    2562 generic.go:358] "Generic (PLEG): container finished" podID="e1367fb0-fb43-4995-9af7-9be8baf478af" containerID="ef421d80a2409ccaa82ed4cc2de79fbafb27dfa2169779bb8fa6d8709a5200c1" exitCode=0
May 25 12:13:21.199161 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:21.199008    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sfdq" event={"ID":"e1367fb0-fb43-4995-9af7-9be8baf478af","Type":"ContainerDied","Data":"ef421d80a2409ccaa82ed4cc2de79fbafb27dfa2169779bb8fa6d8709a5200c1"}
May 25 12:13:21.200754 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:21.200705    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr" event={"ID":"2097934f-3b96-486a-aa25-59074840b23e","Type":"ContainerStarted","Data":"b431f7192b148d8f95cdf8db81d53dd6b603fd550e9daa3fc1d43c5203d47b09"}
May 25 12:13:21.202271 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:21.202202    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" event={"ID":"d7ea3aea-da4a-49d5-be77-4de8dddf91bb","Type":"ContainerStarted","Data":"ca1e84a2c78359941cc8ec8f1617754fb1e51ef1c403568d066b00dabc6f42cc"}
May 25 12:13:21.204478 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:21.204292    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v" event={"ID":"06c76375-02e4-4984-a004-7e997033a7f9","Type":"ContainerStarted","Data":"55e6d0a3ddd1e33c1ea25d91998c24db9b5e0ccbc3d00479cd9f0f63b4aef743"}
May 25 12:13:22.217686 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:22.216915    2562 generic.go:358] "Generic (PLEG): container finished" podID="e1367fb0-fb43-4995-9af7-9be8baf478af" containerID="63b25af2974491aee8383ec6ebf3c8bb63305da9345930014387f710357bff10" exitCode=0
May 25 12:13:22.217686 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:22.217113    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sfdq" event={"ID":"e1367fb0-fb43-4995-9af7-9be8baf478af","Type":"ContainerDied","Data":"63b25af2974491aee8383ec6ebf3c8bb63305da9345930014387f710357bff10"}
May 25 12:13:23.019022 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:23.018970    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:23.019210 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:23.019043    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:23.019210 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:23.019103    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:23.019210 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:23.019137    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 12:13:23.019419 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:23.019206    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:23.019419 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:23.019253    2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:23.019419 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:23.019216    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert podName:a1ed9528-15dc-44a5-94ce-b457a0eccece nodeName:}" failed. No retries permitted until 2026-05-25 12:13:27.019193222 +0000 UTC m=+42.677524611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-httbc" (UID: "a1ed9528-15dc-44a5-94ce-b457a0eccece") : secret "networking-console-plugin-cert" not found
May 25 12:13:23.019419 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:23.019284    2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c68f5fcb8-qv8f5: secret "image-registry-tls" not found
May 25 12:13:23.019419 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:23.019303    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert podName:6ebba530-a3c2-4deb-918c-b0f9ca60c101 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:27.019287002 +0000 UTC m=+42.677618399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert") pod "ingress-canary-bc7wq" (UID: "6ebba530-a3c2-4deb-918c-b0f9ca60c101") : secret "canary-serving-cert" not found
May 25 12:13:23.019419 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:23.019325    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls podName:a94f30d8-539a-4618-9075-95c2295727d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:27.019311823 +0000 UTC m=+42.677643212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls") pod "image-registry-5c68f5fcb8-qv8f5" (UID: "a94f30d8-539a-4618-9075-95c2295727d8") : secret "image-registry-tls" not found
May 25 12:13:23.120613 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:23.120576    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:23.120799 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:23.120771    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:23.120868 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:23.120858    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls podName:059085b8-4425-4d8c-b229-bd2bce5c1657 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:27.120834951 +0000 UTC m=+42.779166345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls") pod "dns-default-c77cg" (UID: "059085b8-4425-4d8c-b229-bd2bce5c1657") : secret "dns-default-metrics-tls" not found
May 25 12:13:27.054741 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:27.054699    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:27.055189 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:27.054813    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:27.055189 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:27.054856    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:27.055189 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:27.054863    2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:27.055189 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:27.054882    2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c68f5fcb8-qv8f5: secret "image-registry-tls" not found
May 25 12:13:27.055189 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:27.054939    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls podName:a94f30d8-539a-4618-9075-95c2295727d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:35.054920755 +0000 UTC m=+50.713252156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls") pod "image-registry-5c68f5fcb8-qv8f5" (UID: "a94f30d8-539a-4618-9075-95c2295727d8") : secret "image-registry-tls" not found
May 25 12:13:27.055189 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:27.054960    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:27.055189 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:27.054970    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 12:13:27.055189 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:27.054993    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert podName:6ebba530-a3c2-4deb-918c-b0f9ca60c101 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:35.054980824 +0000 UTC m=+50.713312207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert") pod "ingress-canary-bc7wq" (UID: "6ebba530-a3c2-4deb-918c-b0f9ca60c101") : secret "canary-serving-cert" not found
May 25 12:13:27.055189 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:27.055035    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert podName:a1ed9528-15dc-44a5-94ce-b457a0eccece nodeName:}" failed. No retries permitted until 2026-05-25 12:13:35.055019549 +0000 UTC m=+50.713350935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-httbc" (UID: "a1ed9528-15dc-44a5-94ce-b457a0eccece") : secret "networking-console-plugin-cert" not found
May 25 12:13:27.156145 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:27.156103    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:27.156366 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:27.156303    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:27.156440 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:27.156387    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls podName:059085b8-4425-4d8c-b229-bd2bce5c1657 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:35.156364557 +0000 UTC m=+50.814695945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls") pod "dns-default-c77cg" (UID: "059085b8-4425-4d8c-b229-bd2bce5c1657") : secret "dns-default-metrics-tls" not found
May 25 12:13:28.233253 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:28.233221    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sfdq" event={"ID":"e1367fb0-fb43-4995-9af7-9be8baf478af","Type":"ContainerStarted","Data":"c56ff020f61855fbdec7fb50c3e694fe0949d0edd108a2bcfe6f87115f89abd9"}
May 25 12:13:28.255216 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:28.255174    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7sfdq" podStartSLOduration=11.152157607 podStartE2EDuration="44.255160571s" podCreationTimestamp="2026-05-25 12:12:44 +0000 UTC" firstStartedPulling="2026-05-25 12:12:47.464770598 +0000 UTC m=+3.123101982" lastFinishedPulling="2026-05-25 12:13:20.567773562 +0000 UTC m=+36.226104946" observedRunningTime="2026-05-25 12:13:28.254003673 +0000 UTC m=+43.912335077" watchObservedRunningTime="2026-05-25 12:13:28.255160571 +0000 UTC m=+43.913491968"
May 25 12:13:29.237069 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:29.237030    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr" event={"ID":"2097934f-3b96-486a-aa25-59074840b23e","Type":"ContainerStarted","Data":"ea6740643d0734bbfd8b28eec1b0ee7ee5f3796fe8a1e80fee0eb245eb2c6237"}
May 25 12:13:29.237539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:29.237309    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:29.238529 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:29.238498    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" event={"ID":"d7ea3aea-da4a-49d5-be77-4de8dddf91bb","Type":"ContainerStarted","Data":"7fbc8c8d52982c05bf3b1126404357589432c11dad3eac4a191d9add79e03a53"}
May 25 12:13:29.239301 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:29.239279    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:13:29.240178 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:29.240040    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v" event={"ID":"06c76375-02e4-4984-a004-7e997033a7f9","Type":"ContainerStarted","Data":"3ec5b65f73c4588e421fd624a2bed11eb3a7db593d76af30b05bdcf8268184ba"}
May 25 12:13:29.253018 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:29.252971    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr" podStartSLOduration=30.670082762 podStartE2EDuration="38.25295377s" podCreationTimestamp="2026-05-25 12:12:51 +0000 UTC" firstStartedPulling="2026-05-25 12:13:20.543564127 +0000 UTC m=+36.201895512" lastFinishedPulling="2026-05-25 12:13:28.126435122 +0000 UTC m=+43.784766520" observedRunningTime="2026-05-25 12:13:29.252380292 +0000 UTC m=+44.910711711" watchObservedRunningTime="2026-05-25 12:13:29.25295377 +0000 UTC m=+44.911285176"
May 25 12:13:29.280212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:29.280156    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v" podStartSLOduration=30.696830683 podStartE2EDuration="38.280142661s" podCreationTimestamp="2026-05-25 12:12:51 +0000 UTC" firstStartedPulling="2026-05-25 12:13:20.543741196 +0000 UTC m=+36.202072581" lastFinishedPulling="2026-05-25 12:13:28.127053174 +0000 UTC m=+43.785384559" observedRunningTime="2026-05-25 12:13:29.279793246 +0000 UTC m=+44.938124652" watchObservedRunningTime="2026-05-25 12:13:29.280142661 +0000 UTC m=+44.938474058"
May 25 12:13:31.245366 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:31.245330    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" event={"ID":"d7ea3aea-da4a-49d5-be77-4de8dddf91bb","Type":"ContainerStarted","Data":"dbdb094c5b2201370ac96031057f837b7a5f471baf615874063d684f5acc5e08"}
May 25 12:13:31.245366 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:31.245370    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" event={"ID":"d7ea3aea-da4a-49d5-be77-4de8dddf91bb","Type":"ContainerStarted","Data":"2d0afb1d2af7e5a836049131d4471089fc13eaa2f1ff23cae2b3176992858a42"}
May 25 12:13:31.264580 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:31.264535    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" podStartSLOduration=30.340796729 podStartE2EDuration="40.264523048s" podCreationTimestamp="2026-05-25 12:12:51 +0000 UTC" firstStartedPulling="2026-05-25 12:13:20.543576496 +0000 UTC m=+36.201907880" lastFinishedPulling="2026-05-25 12:13:30.467302799 +0000 UTC m=+46.125634199" observedRunningTime="2026-05-25 12:13:31.264288501 +0000 UTC m=+46.922619906" watchObservedRunningTime="2026-05-25 12:13:31.264523048 +0000 UTC m=+46.922854452"
May 25 12:13:35.120392 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:35.120351    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:35.120860 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:35.120507    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:35.120860 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:35.120529    2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:35.120860 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:35.120551    2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c68f5fcb8-qv8f5: secret "image-registry-tls" not found
May 25 12:13:35.120860 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:35.120601    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:35.120860 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:35.120607    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls podName:a94f30d8-539a-4618-9075-95c2295727d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:51.120591045 +0000 UTC m=+66.778922428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls") pod "image-registry-5c68f5fcb8-qv8f5" (UID: "a94f30d8-539a-4618-9075-95c2295727d8") : secret "image-registry-tls" not found
May 25 12:13:35.120860 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:35.120632    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 12:13:35.120860 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:35.120538    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:35.120860 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:35.120639    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert podName:6ebba530-a3c2-4deb-918c-b0f9ca60c101 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:51.120628217 +0000 UTC m=+66.778959600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert") pod "ingress-canary-bc7wq" (UID: "6ebba530-a3c2-4deb-918c-b0f9ca60c101") : secret "canary-serving-cert" not found
May 25 12:13:35.120860 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:35.120702    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert podName:a1ed9528-15dc-44a5-94ce-b457a0eccece nodeName:}" failed. No retries permitted until 2026-05-25 12:13:51.120690375 +0000 UTC m=+66.779021757 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-httbc" (UID: "a1ed9528-15dc-44a5-94ce-b457a0eccece") : secret "networking-console-plugin-cert" not found
May 25 12:13:35.221212 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:35.221174    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:35.221362 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:35.221317    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:35.221430 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:35.221369    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls podName:059085b8-4425-4d8c-b229-bd2bce5c1657 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:51.221356592 +0000 UTC m=+66.879687974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls") pod "dns-default-c77cg" (UID: "059085b8-4425-4d8c-b229-bd2bce5c1657") : secret "dns-default-metrics-tls" not found
May 25 12:13:45.194762 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:45.194729    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrrrl"
May 25 12:13:50.639328 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:50.639289    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:13:50.642307 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:50.642286    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:13:50.649645 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:50.649627    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 12:13:50.649725 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:50.649699    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs podName:037bfb47-48b6-4248-9537-b2da35bdf80f nodeName:}" failed. No retries permitted until 2026-05-25 12:14:54.649676801 +0000 UTC m=+130.308008203 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs") pod "network-metrics-daemon-22jr7" (UID: "037bfb47-48b6-4248-9537-b2da35bdf80f") : secret "metrics-daemon-secret" not found
May 25 12:13:50.740247 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:50.740207    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:50.743197 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:50.743179    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:13:50.753818 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:50.753800    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:13:50.766112 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:50.766084    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dj7\" (UniqueName: \"kubernetes.io/projected/0a521353-c135-45c9-a6f1-0d36c801505d-kube-api-access-h2dj7\") pod \"network-check-target-rvhv8\" (UID: \"0a521353-c135-45c9-a6f1-0d36c801505d\") " pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:50.974937 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:50.974914    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxdt4\""
May 25 12:13:50.982143 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:50.982122    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:51.100089 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.100061    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rvhv8"]
May 25 12:13:51.103542 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:13:51.103514    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a521353_c135_45c9_a6f1_0d36c801505d.slice/crio-898f2f8449291a8c2f0bb1cd23927c66fb642f675248ce28ff717b842eaf1468 WatchSource:0}: Error finding container 898f2f8449291a8c2f0bb1cd23927c66fb642f675248ce28ff717b842eaf1468: Status 404 returned error can't find the container with id 898f2f8449291a8c2f0bb1cd23927c66fb642f675248ce28ff717b842eaf1468
May 25 12:13:51.143958 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.143928    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:13:51.144115 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.143976    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:13:51.144115 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.144029    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:13:51.144211 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:51.144108    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 12:13:51.144211 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:51.144153    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:51.144211 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:51.144174    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert podName:a1ed9528-15dc-44a5-94ce-b457a0eccece nodeName:}" failed. No retries permitted until 2026-05-25 12:14:23.144158678 +0000 UTC m=+98.802490062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-httbc" (UID: "a1ed9528-15dc-44a5-94ce-b457a0eccece") : secret "networking-console-plugin-cert" not found
May 25 12:13:51.144211 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:51.144198    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert podName:6ebba530-a3c2-4deb-918c-b0f9ca60c101 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:23.144185722 +0000 UTC m=+98.802517104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert") pod "ingress-canary-bc7wq" (UID: "6ebba530-a3c2-4deb-918c-b0f9ca60c101") : secret "canary-serving-cert" not found
May 25 12:13:51.144373 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:51.144211    2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:51.144373 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:51.144229    2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c68f5fcb8-qv8f5: secret "image-registry-tls" not found
May 25 12:13:51.144373 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:51.144306    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls podName:a94f30d8-539a-4618-9075-95c2295727d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:23.144290057 +0000 UTC m=+98.802621457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls") pod "image-registry-5c68f5fcb8-qv8f5" (UID: "a94f30d8-539a-4618-9075-95c2295727d8") : secret "image-registry-tls" not found
May 25 12:13:51.245542 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.245465    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:13:51.245670 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:51.245581    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:51.245670 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:13:51.245630    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls podName:059085b8-4425-4d8c-b229-bd2bce5c1657 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:23.24561733 +0000 UTC m=+98.903948713 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls") pod "dns-default-c77cg" (UID: "059085b8-4425-4d8c-b229-bd2bce5c1657") : secret "dns-default-metrics-tls" not found
May 25 12:13:51.284696 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.284661    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rvhv8" event={"ID":"0a521353-c135-45c9-a6f1-0d36c801505d","Type":"ContainerStarted","Data":"898f2f8449291a8c2f0bb1cd23927c66fb642f675248ce28ff717b842eaf1468"}
May 25 12:13:51.648180 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.648086    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:51.651074 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.651052    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 12:13:51.661049 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.661022    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f71c2f0-743b-4295-af1d-2182812eaeb6-original-pull-secret\") pod \"global-pull-secret-syncer-wlsj7\" (UID: \"3f71c2f0-743b-4295-af1d-2182812eaeb6\") " pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:51.861022 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:51.860985    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wlsj7"
May 25 12:13:52.004983 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:52.004735    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wlsj7"]
May 25 12:13:52.007389 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:13:52.007360    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f71c2f0_743b_4295_af1d_2182812eaeb6.slice/crio-5e37d6db51da2ec195ad524f07fcad2d49f93e244b8edeb8f3765544d90737da WatchSource:0}: Error finding container 5e37d6db51da2ec195ad524f07fcad2d49f93e244b8edeb8f3765544d90737da: Status 404 returned error can't find the container with id 5e37d6db51da2ec195ad524f07fcad2d49f93e244b8edeb8f3765544d90737da
May 25 12:13:52.287927 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:52.287888    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wlsj7" event={"ID":"3f71c2f0-743b-4295-af1d-2182812eaeb6","Type":"ContainerStarted","Data":"5e37d6db51da2ec195ad524f07fcad2d49f93e244b8edeb8f3765544d90737da"}
May 25 12:13:56.300009 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:56.299970    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rvhv8" event={"ID":"0a521353-c135-45c9-a6f1-0d36c801505d","Type":"ContainerStarted","Data":"692f1621f8356dde8ce51335fb63174ae7252fc25f286e3b744d7d13bf073079"}
May 25 12:13:56.300553 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:56.300063    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:13:56.301286 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:56.301251    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wlsj7" event={"ID":"3f71c2f0-743b-4295-af1d-2182812eaeb6","Type":"ContainerStarted","Data":"7060f925737e749c6c42a5ec27d7a872c94ccbab609e2488ffbab2f1e6b43288"}
May 25 12:13:56.314526 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:56.314478    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rvhv8" podStartSLOduration=66.734634623 podStartE2EDuration="1m11.314462796s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="2026-05-25 12:13:51.10597616 +0000 UTC m=+66.764307558" lastFinishedPulling="2026-05-25 12:13:55.685804349 +0000 UTC m=+71.344135731" observedRunningTime="2026-05-25 12:13:56.31373475 +0000 UTC m=+71.972066155" watchObservedRunningTime="2026-05-25 12:13:56.314462796 +0000 UTC m=+71.972794202"
May 25 12:13:56.330185 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:13:56.330136    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wlsj7" podStartSLOduration=65.64993662 podStartE2EDuration="1m9.330122351s" podCreationTimestamp="2026-05-25 12:12:47 +0000 UTC" firstStartedPulling="2026-05-25 12:13:52.0093799 +0000 UTC m=+67.667711287" lastFinishedPulling="2026-05-25 12:13:55.689565632 +0000 UTC m=+71.347897018" observedRunningTime="2026-05-25 12:13:56.328948921 +0000 UTC m=+71.987280325" watchObservedRunningTime="2026-05-25 12:13:56.330122351 +0000 UTC m=+71.988453753"
May 25 12:14:23.207922 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:14:23.207881    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:14:23.208360 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:14:23.207929    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:14:23.208360 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:14:23.207982    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:14:23.208360 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:23.208040    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 12:14:23.208360 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:23.208097    2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:14:23.208360 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:23.208108    2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c68f5fcb8-qv8f5: secret "image-registry-tls" not found
May 25 12:14:23.208360 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:23.208112    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert podName:a1ed9528-15dc-44a5-94ce-b457a0eccece nodeName:}" failed. No retries permitted until 2026-05-25 12:15:27.208089734 +0000 UTC m=+162.866421128 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-httbc" (UID: "a1ed9528-15dc-44a5-94ce-b457a0eccece") : secret "networking-console-plugin-cert" not found
May 25 12:14:23.208360 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:23.208118    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:14:23.208360 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:23.208150    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls podName:a94f30d8-539a-4618-9075-95c2295727d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:27.208139214 +0000 UTC m=+162.866470597 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls") pod "image-registry-5c68f5fcb8-qv8f5" (UID: "a94f30d8-539a-4618-9075-95c2295727d8") : secret "image-registry-tls" not found
May 25 12:14:23.208360 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:23.208181    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert podName:6ebba530-a3c2-4deb-918c-b0f9ca60c101 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:27.20816465 +0000 UTC m=+162.866496033 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert") pod "ingress-canary-bc7wq" (UID: "6ebba530-a3c2-4deb-918c-b0f9ca60c101") : secret "canary-serving-cert" not found
May 25 12:14:23.308581 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:14:23.308543    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:14:23.308742 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:23.308678    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:14:23.308742 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:23.308737    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls podName:059085b8-4425-4d8c-b229-bd2bce5c1657 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:27.308722928 +0000 UTC m=+162.967054311 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls") pod "dns-default-c77cg" (UID: "059085b8-4425-4d8c-b229-bd2bce5c1657") : secret "dns-default-metrics-tls" not found
May 25 12:14:27.306619 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:14:27.306581    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rvhv8"
May 25 12:14:54.748707 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:14:54.748663    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:14:54.749204 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:54.748833    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 12:14:54.749204 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:14:54.748943    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs podName:037bfb47-48b6-4248-9537-b2da35bdf80f nodeName:}" failed. No retries permitted until 2026-05-25 12:16:56.748905757 +0000 UTC m=+252.407237145 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs") pod "network-metrics-daemon-22jr7" (UID: "037bfb47-48b6-4248-9537-b2da35bdf80f") : secret "metrics-daemon-secret" not found
May 25 12:15:17.600802 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:17.600771    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9q65_01253226-6d29-4eac-9a58-3d9a8be0032d/dns-node-resolver/0.log"
May 25 12:15:18.400556 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:18.400528    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cf9x4_cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f/node-ca/0.log"
May 25 12:15:22.236103 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:15:22.236056    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5" podUID="a94f30d8-539a-4618-9075-95c2295727d8"
May 25 12:15:22.255366 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:15:22.255332    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc" podUID="a1ed9528-15dc-44a5-94ce-b457a0eccece"
May 25 12:15:22.322312 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:15:22.322252    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-bc7wq" podUID="6ebba530-a3c2-4deb-918c-b0f9ca60c101"
May 25 12:15:22.331423 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:15:22.331396    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-c77cg" podUID="059085b8-4425-4d8c-b229-bd2bce5c1657"
May 25 12:15:22.499956 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:22.499880    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:15:22.499956 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:22.499919    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:15:22.500108 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:22.500051    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:15:23.967279 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:15:23.967211    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-22jr7" podUID="037bfb47-48b6-4248-9537-b2da35bdf80f"
May 25 12:15:27.300113 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.300073    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:15:27.300113 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.300113    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:15:27.300586 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.300149    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:15:27.300586 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:15:27.300218    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 12:15:27.300586 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:15:27.300307    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert podName:a1ed9528-15dc-44a5-94ce-b457a0eccece nodeName:}" failed. No retries permitted until 2026-05-25 12:17:29.300290142 +0000 UTC m=+284.958621529 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-httbc" (UID: "a1ed9528-15dc-44a5-94ce-b457a0eccece") : secret "networking-console-plugin-cert" not found
May 25 12:15:27.302539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.302518    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"image-registry-5c68f5fcb8-qv8f5\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") " pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:15:27.302656 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.302525    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebba530-a3c2-4deb-918c-b0f9ca60c101-cert\") pod \"ingress-canary-bc7wq\" (UID: \"6ebba530-a3c2-4deb-918c-b0f9ca60c101\") " pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:15:27.401153 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.401117    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:15:27.403482 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.403453    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059085b8-4425-4d8c-b229-bd2bce5c1657-metrics-tls\") pod \"dns-default-c77cg\" (UID: \"059085b8-4425-4d8c-b229-bd2bce5c1657\") " pod="openshift-dns/dns-default-c77cg"
May 25 12:15:27.604399 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.604310    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zkb72\""
May 25 12:15:27.604399 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.604310    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k9dhz\""
May 25 12:15:27.611536 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.611508    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:15:27.611536 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.611526    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bc7wq"
May 25 12:15:27.743316 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.743284    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bc7wq"]
May 25 12:15:27.746914 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:15:27.746871    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebba530_a3c2_4deb_918c_b0f9ca60c101.slice/crio-42772567324da54f370562755c8175b53b591798e83a04028db466ac8a06de58 WatchSource:0}: Error finding container 42772567324da54f370562755c8175b53b591798e83a04028db466ac8a06de58: Status 404 returned error can't find the container with id 42772567324da54f370562755c8175b53b591798e83a04028db466ac8a06de58
May 25 12:15:27.759182 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:27.759159    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"]
May 25 12:15:27.761914 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:15:27.761890    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94f30d8_539a_4618_9075_95c2295727d8.slice/crio-8daa5798a792ef0a2b8a3e8049c82b7363429615148eb3ee441cc2ddffb91a96 WatchSource:0}: Error finding container 8daa5798a792ef0a2b8a3e8049c82b7363429615148eb3ee441cc2ddffb91a96: Status 404 returned error can't find the container with id 8daa5798a792ef0a2b8a3e8049c82b7363429615148eb3ee441cc2ddffb91a96
May 25 12:15:28.520552 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.520511    2562 generic.go:358] "Generic (PLEG): container finished" podID="2097934f-3b96-486a-aa25-59074840b23e" containerID="ea6740643d0734bbfd8b28eec1b0ee7ee5f3796fe8a1e80fee0eb245eb2c6237" exitCode=1
May 25 12:15:28.521012 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.520601    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr" event={"ID":"2097934f-3b96-486a-aa25-59074840b23e","Type":"ContainerDied","Data":"ea6740643d0734bbfd8b28eec1b0ee7ee5f3796fe8a1e80fee0eb245eb2c6237"}
May 25 12:15:28.521012 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.521000    2562 scope.go:117] "RemoveContainer" containerID="ea6740643d0734bbfd8b28eec1b0ee7ee5f3796fe8a1e80fee0eb245eb2c6237"
May 25 12:15:28.522145 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.522105    2562 generic.go:358] "Generic (PLEG): container finished" podID="06c76375-02e4-4984-a004-7e997033a7f9" containerID="3ec5b65f73c4588e421fd624a2bed11eb3a7db593d76af30b05bdcf8268184ba" exitCode=255
May 25 12:15:28.522495 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.522139    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v" event={"ID":"06c76375-02e4-4984-a004-7e997033a7f9","Type":"ContainerDied","Data":"3ec5b65f73c4588e421fd624a2bed11eb3a7db593d76af30b05bdcf8268184ba"}
May 25 12:15:28.522760 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.522748    2562 scope.go:117] "RemoveContainer" containerID="3ec5b65f73c4588e421fd624a2bed11eb3a7db593d76af30b05bdcf8268184ba"
May 25 12:15:28.524481 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.524454    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5" event={"ID":"a94f30d8-539a-4618-9075-95c2295727d8","Type":"ContainerStarted","Data":"8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca"}
May 25 12:15:28.524578 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.524492    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5" event={"ID":"a94f30d8-539a-4618-9075-95c2295727d8","Type":"ContainerStarted","Data":"8daa5798a792ef0a2b8a3e8049c82b7363429615148eb3ee441cc2ddffb91a96"}
May 25 12:15:28.524635 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.524608    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:15:28.525691 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.525673    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bc7wq" event={"ID":"6ebba530-a3c2-4deb-918c-b0f9ca60c101","Type":"ContainerStarted","Data":"42772567324da54f370562755c8175b53b591798e83a04028db466ac8a06de58"}
May 25 12:15:28.554080 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:28.554028    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5" podStartSLOduration=163.554007814 podStartE2EDuration="2m43.554007814s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:15:28.552969741 +0000 UTC m=+164.211301169" watchObservedRunningTime="2026-05-25 12:15:28.554007814 +0000 UTC m=+164.212339220"
May 25 12:15:29.237241 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:29.237207    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:15:29.523509 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:29.523483    2562 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:15:29.529406 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:29.529373    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-85b9c949c6-cqn2v" event={"ID":"06c76375-02e4-4984-a004-7e997033a7f9","Type":"ContainerStarted","Data":"c748ad317992f427007831fe983f429184ee8ba240073894a32f7357eee6bef6"}
May 25 12:15:29.531976 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:29.531955    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr" event={"ID":"2097934f-3b96-486a-aa25-59074840b23e","Type":"ContainerStarted","Data":"44ff6bd464ec56ec8b451ad3a4211ead5e54e9539503eb492e3abe67266846de"}
May 25 12:15:30.538868 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:30.538832    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bc7wq" event={"ID":"6ebba530-a3c2-4deb-918c-b0f9ca60c101","Type":"ContainerStarted","Data":"ae957050f56ee2cc627e3b5b5be003207d7ebc425dba115a56ded02102e52c1b"}
May 25 12:15:30.539359 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:30.539058    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:15:30.540412 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:30.540393    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c87bf6ff-24zbr"
May 25 12:15:30.555820 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:30.555773    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bc7wq" podStartSLOduration=129.824111989 podStartE2EDuration="2m11.555762634s" podCreationTimestamp="2026-05-25 12:13:19 +0000 UTC" firstStartedPulling="2026-05-25 12:15:27.748820859 +0000 UTC m=+163.407152242" lastFinishedPulling="2026-05-25 12:15:29.48047149 +0000 UTC m=+165.138802887" observedRunningTime="2026-05-25 12:15:30.554919621 +0000 UTC m=+166.213251029" watchObservedRunningTime="2026-05-25 12:15:30.555762634 +0000 UTC m=+166.214094032"
May 25 12:15:36.783245 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.783144    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f5nx7"]
May 25 12:15:36.785200 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.785181    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.789144 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.789124    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 12:15:36.789144 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.789126    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m27gk\""
May 25 12:15:36.789333 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.789126    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 12:15:36.789797 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.789780    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 12:15:36.792106 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.792087    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 12:15:36.800790 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.800762    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5nx7"]
May 25 12:15:36.872105 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.872067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4818e986-b543-4e21-b438-69b2f3003855-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.872290 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.872117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvb4p\" (UniqueName: \"kubernetes.io/projected/4818e986-b543-4e21-b438-69b2f3003855-kube-api-access-fvb4p\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.872290 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.872178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4818e986-b543-4e21-b438-69b2f3003855-data-volume\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.872290 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.872232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4818e986-b543-4e21-b438-69b2f3003855-crio-socket\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.872290 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.872277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4818e986-b543-4e21-b438-69b2f3003855-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.950194 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.950161    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c77cg"
May 25 12:15:36.953003 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.952983    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fp9z9\""
May 25 12:15:36.961547 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.961528    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c77cg"
May 25 12:15:36.973591 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.973566    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4818e986-b543-4e21-b438-69b2f3003855-data-volume\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.973707 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.973602    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4818e986-b543-4e21-b438-69b2f3003855-crio-socket\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.973707 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.973629    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4818e986-b543-4e21-b438-69b2f3003855-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.973786 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.973738    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4818e986-b543-4e21-b438-69b2f3003855-crio-socket\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.973851 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.973832    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4818e986-b543-4e21-b438-69b2f3003855-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.973916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.973899    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fvb4p\" (UniqueName: \"kubernetes.io/projected/4818e986-b543-4e21-b438-69b2f3003855-kube-api-access-fvb4p\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.973962 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.973920    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4818e986-b543-4e21-b438-69b2f3003855-data-volume\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.974147 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.974130    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4818e986-b543-4e21-b438-69b2f3003855-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.976334 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.976286    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4818e986-b543-4e21-b438-69b2f3003855-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:36.983233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:36.983206    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvb4p\" (UniqueName: \"kubernetes.io/projected/4818e986-b543-4e21-b438-69b2f3003855-kube-api-access-fvb4p\") pod \"insights-runtime-extractor-f5nx7\" (UID: \"4818e986-b543-4e21-b438-69b2f3003855\") " pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:37.085891 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:37.085858    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c77cg"]
May 25 12:15:37.089090 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:15:37.089062    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059085b8_4425_4d8c_b229_bd2bce5c1657.slice/crio-a423176c8b3bffe40c2ec983900a535524c204cd3286177f992401517bb3497f WatchSource:0}: Error finding container a423176c8b3bffe40c2ec983900a535524c204cd3286177f992401517bb3497f: Status 404 returned error can't find the container with id a423176c8b3bffe40c2ec983900a535524c204cd3286177f992401517bb3497f
May 25 12:15:37.094355 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:37.094333    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5nx7"
May 25 12:15:37.216666 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:37.216632    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5nx7"]
May 25 12:15:37.219585 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:15:37.219549    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4818e986_b543_4e21_b438_69b2f3003855.slice/crio-dd583b3b6da4cc04f4c1cf1b51b45a45fb7e39f776ad5216071cae8e1a4e3db5 WatchSource:0}: Error finding container dd583b3b6da4cc04f4c1cf1b51b45a45fb7e39f776ad5216071cae8e1a4e3db5: Status 404 returned error can't find the container with id dd583b3b6da4cc04f4c1cf1b51b45a45fb7e39f776ad5216071cae8e1a4e3db5
May 25 12:15:37.557881 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:37.557844    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5nx7" event={"ID":"4818e986-b543-4e21-b438-69b2f3003855","Type":"ContainerStarted","Data":"7769de6dfa21d5824d64ffac3dbac35b6b17ef2bac9ec58948ca02cfb2995f70"}
May 25 12:15:37.557881 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:37.557887    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5nx7" event={"ID":"4818e986-b543-4e21-b438-69b2f3003855","Type":"ContainerStarted","Data":"dd583b3b6da4cc04f4c1cf1b51b45a45fb7e39f776ad5216071cae8e1a4e3db5"}
May 25 12:15:37.559083 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:37.559054    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c77cg" event={"ID":"059085b8-4425-4d8c-b229-bd2bce5c1657","Type":"ContainerStarted","Data":"a423176c8b3bffe40c2ec983900a535524c204cd3286177f992401517bb3497f"}
May 25 12:15:38.563290 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:38.563190    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5nx7" event={"ID":"4818e986-b543-4e21-b438-69b2f3003855","Type":"ContainerStarted","Data":"45c65e8a7ffcffda7bd841b77fb79c0f833236bb40803cc3b012bd6f7ee7a6f4"}
May 25 12:15:38.564612 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:38.564591    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c77cg" event={"ID":"059085b8-4425-4d8c-b229-bd2bce5c1657","Type":"ContainerStarted","Data":"7148bee2e41a77405d74d5e92da716b8768d8f9ca60d0b71686a9ed352e76b86"}
May 25 12:15:38.564612 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:38.564619    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c77cg" event={"ID":"059085b8-4425-4d8c-b229-bd2bce5c1657","Type":"ContainerStarted","Data":"d98b9421e4a64a398d952eb3af60a5c53a7532172d0eb4af400af96dfb6d23da"}
May 25 12:15:38.564756 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:38.564747    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c77cg"
May 25 12:15:38.580399 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:38.580353    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-c77cg" podStartSLOduration=138.358265474 podStartE2EDuration="2m19.580339521s" podCreationTimestamp="2026-05-25 12:13:19 +0000 UTC" firstStartedPulling="2026-05-25 12:15:37.090978795 +0000 UTC m=+172.749310191" lastFinishedPulling="2026-05-25 12:15:38.31305284 +0000 UTC m=+173.971384238" observedRunningTime="2026-05-25 12:15:38.579810856 +0000 UTC m=+174.238142262" watchObservedRunningTime="2026-05-25 12:15:38.580339521 +0000 UTC m=+174.238670935"
May 25 12:15:38.950922 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:38.950841    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:15:39.570173 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:39.570143    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5nx7" event={"ID":"4818e986-b543-4e21-b438-69b2f3003855","Type":"ContainerStarted","Data":"9fe08dfea83906125a53604900f36571182e27c39632010a0e819a0515133d31"}
May 25 12:15:39.588139 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:39.588093    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f5nx7" podStartSLOduration=1.352192392 podStartE2EDuration="3.588079011s" podCreationTimestamp="2026-05-25 12:15:36 +0000 UTC" firstStartedPulling="2026-05-25 12:15:37.272776359 +0000 UTC m=+172.931107746" lastFinishedPulling="2026-05-25 12:15:39.50866298 +0000 UTC m=+175.166994365" observedRunningTime="2026-05-25 12:15:39.586582211 +0000 UTC m=+175.244913615" watchObservedRunningTime="2026-05-25 12:15:39.588079011 +0000 UTC m=+175.246410417"
May 25 12:15:47.615392 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:47.615350    2562 patch_prober.go:28] interesting pod/image-registry-5c68f5fcb8-qv8f5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 12:15:47.615809 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:47.615407    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5" podUID="a94f30d8-539a-4618-9075-95c2295727d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:15:48.572282 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:48.572235    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c77cg"
May 25 12:15:49.535779 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:49.535752    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:15:58.649835 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.649804    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sh72p"]
May 25 12:15:58.652805 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.652785    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.655405 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.655379    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qb5mz\""
May 25 12:15:58.655587 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.655428    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 12:15:58.655587 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.655486    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 12:15:58.656660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.656621    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 12:15:58.656775 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.656683    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 12:15:58.656775 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.656695    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 12:15:58.657041 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.657028    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 12:15:58.841899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.841864    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-wtmp\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.842090 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.841919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjfdq\" (UniqueName: \"kubernetes.io/projected/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-kube-api-access-hjfdq\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.842090 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.841939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.842090 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.841986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-tls\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.842090 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.842010    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-root\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.842090 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.842025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-sys\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.842090 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.842039    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-metrics-client-ca\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.842090 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.842059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-textfile\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.842381 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.842123    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943521 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943427    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-tls\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943521 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943468    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-root\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943521 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943485    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-sys\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943521 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943500    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-metrics-client-ca\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943521 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943523    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-textfile\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943538    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-root\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943556    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943561    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-sys\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:15:58.943590    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943623    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-wtmp\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:15:58.943669    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-tls podName:c14ae715-c96a-43de-9e7b-6c9fe09ce2fd nodeName:}" failed. No retries permitted until 2026-05-25 12:15:59.443646555 +0000 UTC m=+195.101977944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-tls") pod "node-exporter-sh72p" (UID: "c14ae715-c96a-43de-9e7b-6c9fe09ce2fd") : secret "node-exporter-tls" not found
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943692    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hjfdq\" (UniqueName: \"kubernetes.io/projected/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-kube-api-access-hjfdq\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943729    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943761    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-wtmp\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.943893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.943872    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-textfile\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.944207 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.944160    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-metrics-client-ca\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.944274 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.944240    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.945900 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.945881    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:58.952065 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:58.952042    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjfdq\" (UniqueName: \"kubernetes.io/projected/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-kube-api-access-hjfdq\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:59.066657 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:59.066622    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"]
May 25 12:15:59.449122 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:59.449083    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-tls\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:59.451365 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:59.451336    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c14ae715-c96a-43de-9e7b-6c9fe09ce2fd-node-exporter-tls\") pod \"node-exporter-sh72p\" (UID: \"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd\") " pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:59.561559 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:59.561526    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sh72p"
May 25 12:15:59.569328 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:15:59.569291    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14ae715_c96a_43de_9e7b_6c9fe09ce2fd.slice/crio-23c833eb6d586e2127e8a36d911473f1c5c27bf4a88a6115b5f616e0f96032b0 WatchSource:0}: Error finding container 23c833eb6d586e2127e8a36d911473f1c5c27bf4a88a6115b5f616e0f96032b0: Status 404 returned error can't find the container with id 23c833eb6d586e2127e8a36d911473f1c5c27bf4a88a6115b5f616e0f96032b0
May 25 12:15:59.621030 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:15:59.620981    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sh72p" event={"ID":"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd","Type":"ContainerStarted","Data":"23c833eb6d586e2127e8a36d911473f1c5c27bf4a88a6115b5f616e0f96032b0"}
May 25 12:16:00.625480 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:00.625442    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sh72p" event={"ID":"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd","Type":"ContainerStarted","Data":"749ed8992a5b05016b1c4a777198e338d1cbd828f401fa23bcba0c354ebd9ab4"}
May 25 12:16:01.629207 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:01.629167    2562 generic.go:358] "Generic (PLEG): container finished" podID="c14ae715-c96a-43de-9e7b-6c9fe09ce2fd" containerID="749ed8992a5b05016b1c4a777198e338d1cbd828f401fa23bcba0c354ebd9ab4" exitCode=0
May 25 12:16:01.629585 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:01.629234    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sh72p" event={"ID":"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd","Type":"ContainerDied","Data":"749ed8992a5b05016b1c4a777198e338d1cbd828f401fa23bcba0c354ebd9ab4"}
May 25 12:16:02.633842 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:02.633809    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sh72p" event={"ID":"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd","Type":"ContainerStarted","Data":"21abe519fe166589a72a93c1a9ca99c39c62b06a2db304b7af412f4f7b2b8653"}
May 25 12:16:02.633842 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:02.633847    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sh72p" event={"ID":"c14ae715-c96a-43de-9e7b-6c9fe09ce2fd","Type":"ContainerStarted","Data":"bc90421176a6550685a23dfcddd59312db0de6820ef6ce1d561b0daacb60cc3a"}
May 25 12:16:02.655166 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:02.655119    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sh72p" podStartSLOduration=3.7150526409999998 podStartE2EDuration="4.655104337s" podCreationTimestamp="2026-05-25 12:15:58 +0000 UTC" firstStartedPulling="2026-05-25 12:15:59.57176698 +0000 UTC m=+195.230098363" lastFinishedPulling="2026-05-25 12:16:00.511818677 +0000 UTC m=+196.170150059" observedRunningTime="2026-05-25 12:16:02.654788857 +0000 UTC m=+198.313120261" watchObservedRunningTime="2026-05-25 12:16:02.655104337 +0000 UTC m=+198.313435769"
May 25 12:16:09.591172 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:09.591129    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" podUID="d7ea3aea-da4a-49d5-be77-4de8dddf91bb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 12:16:19.591112 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:19.591074    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" podUID="d7ea3aea-da4a-49d5-be77-4de8dddf91bb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 12:16:19.809583 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:19.809551    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bc7wq_6ebba530-a3c2-4deb-918c-b0f9ca60c101/serve-healthcheck-canary/0.log"
May 25 12:16:24.085190 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.085145    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5" podUID="a94f30d8-539a-4618-9075-95c2295727d8" containerName="registry" containerID="cri-o://8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca" gracePeriod=30
May 25 12:16:24.317250 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.317227    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:16:24.431075 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.430984    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-bound-sa-token\") pod \"a94f30d8-539a-4618-9075-95c2295727d8\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") "
May 25 12:16:24.431075 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.431041    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt82f\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-kube-api-access-dt82f\") pod \"a94f30d8-539a-4618-9075-95c2295727d8\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") "
May 25 12:16:24.431075 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.431065    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a94f30d8-539a-4618-9075-95c2295727d8-ca-trust-extracted\") pod \"a94f30d8-539a-4618-9075-95c2295727d8\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") "
May 25 12:16:24.431362 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.431088    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-installation-pull-secrets\") pod \"a94f30d8-539a-4618-9075-95c2295727d8\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") "
May 25 12:16:24.431362 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.431121    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-image-registry-private-configuration\") pod \"a94f30d8-539a-4618-9075-95c2295727d8\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") "
May 25 12:16:24.431362 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.431137    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") pod \"a94f30d8-539a-4618-9075-95c2295727d8\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") "
May 25 12:16:24.431362 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.431166    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-registry-certificates\") pod \"a94f30d8-539a-4618-9075-95c2295727d8\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") "
May 25 12:16:24.431362 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.431215    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-trusted-ca\") pod \"a94f30d8-539a-4618-9075-95c2295727d8\" (UID: \"a94f30d8-539a-4618-9075-95c2295727d8\") "
May 25 12:16:24.431875 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.431820    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a94f30d8-539a-4618-9075-95c2295727d8" (UID: "a94f30d8-539a-4618-9075-95c2295727d8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:24.431983 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.431882    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a94f30d8-539a-4618-9075-95c2295727d8" (UID: "a94f30d8-539a-4618-9075-95c2295727d8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:24.433910 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.433867    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-kube-api-access-dt82f" (OuterVolumeSpecName: "kube-api-access-dt82f") pod "a94f30d8-539a-4618-9075-95c2295727d8" (UID: "a94f30d8-539a-4618-9075-95c2295727d8"). InnerVolumeSpecName "kube-api-access-dt82f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:24.433910 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.433867    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a94f30d8-539a-4618-9075-95c2295727d8" (UID: "a94f30d8-539a-4618-9075-95c2295727d8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:24.434084 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.433977    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a94f30d8-539a-4618-9075-95c2295727d8" (UID: "a94f30d8-539a-4618-9075-95c2295727d8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:24.434084 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.433973    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a94f30d8-539a-4618-9075-95c2295727d8" (UID: "a94f30d8-539a-4618-9075-95c2295727d8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:24.434084 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.434057    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a94f30d8-539a-4618-9075-95c2295727d8" (UID: "a94f30d8-539a-4618-9075-95c2295727d8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:24.439614 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.439591    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94f30d8-539a-4618-9075-95c2295727d8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a94f30d8-539a-4618-9075-95c2295727d8" (UID: "a94f30d8-539a-4618-9075-95c2295727d8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:16:24.532588 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.532533    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-image-registry-private-configuration\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:16:24.532588 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.532584    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-registry-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:16:24.532588 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.532594    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-registry-certificates\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:16:24.532588 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.532605    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a94f30d8-539a-4618-9075-95c2295727d8-trusted-ca\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:16:24.532845 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.532615    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-bound-sa-token\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:16:24.532845 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.532623    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt82f\" (UniqueName: \"kubernetes.io/projected/a94f30d8-539a-4618-9075-95c2295727d8-kube-api-access-dt82f\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:16:24.532845 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.532632    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a94f30d8-539a-4618-9075-95c2295727d8-ca-trust-extracted\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:16:24.532845 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.532640    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a94f30d8-539a-4618-9075-95c2295727d8-installation-pull-secrets\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:16:24.693409 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.693312    2562 generic.go:358] "Generic (PLEG): container finished" podID="a94f30d8-539a-4618-9075-95c2295727d8" containerID="8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca" exitCode=0
May 25 12:16:24.693409 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.693356    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5" event={"ID":"a94f30d8-539a-4618-9075-95c2295727d8","Type":"ContainerDied","Data":"8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca"}
May 25 12:16:24.693409 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.693373    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"
May 25 12:16:24.693409 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.693382    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c68f5fcb8-qv8f5" event={"ID":"a94f30d8-539a-4618-9075-95c2295727d8","Type":"ContainerDied","Data":"8daa5798a792ef0a2b8a3e8049c82b7363429615148eb3ee441cc2ddffb91a96"}
May 25 12:16:24.693409 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.693402    2562 scope.go:117] "RemoveContainer" containerID="8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca"
May 25 12:16:24.701906 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.701891    2562 scope.go:117] "RemoveContainer" containerID="8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca"
May 25 12:16:24.702179 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:16:24.702158    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca\": container with ID starting with 8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca not found: ID does not exist" containerID="8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca"
May 25 12:16:24.702217 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.702190    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca"} err="failed to get container status \"8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca\": rpc error: code = NotFound desc = could not find container \"8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca\": container with ID starting with 8913e0c44312e24a0e9f0ed58d53fbc0ff88c0a19870684805b2d5ac4e1a26ca not found: ID does not exist"
May 25 12:16:24.713536 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.713513    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"]
May 25 12:16:24.716837 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.716813    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c68f5fcb8-qv8f5"]
May 25 12:16:24.953726 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:24.953646    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a94f30d8-539a-4618-9075-95c2295727d8" path="/var/lib/kubelet/pods/a94f30d8-539a-4618-9075-95c2295727d8/volumes"
May 25 12:16:29.591121 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:29.591083    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" podUID="d7ea3aea-da4a-49d5-be77-4de8dddf91bb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 12:16:29.591543 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:29.591149    2562 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr"
May 25 12:16:29.591628 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:29.591611    2562 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"dbdb094c5b2201370ac96031057f837b7a5f471baf615874063d684f5acc5e08"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 25 12:16:29.591664 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:29.591647    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" podUID="d7ea3aea-da4a-49d5-be77-4de8dddf91bb" containerName="service-proxy" containerID="cri-o://dbdb094c5b2201370ac96031057f837b7a5f471baf615874063d684f5acc5e08" gracePeriod=30
May 25 12:16:29.707948 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:29.707807    2562 generic.go:358] "Generic (PLEG): container finished" podID="d7ea3aea-da4a-49d5-be77-4de8dddf91bb" containerID="dbdb094c5b2201370ac96031057f837b7a5f471baf615874063d684f5acc5e08" exitCode=2
May 25 12:16:29.707948 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:29.707855    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" event={"ID":"d7ea3aea-da4a-49d5-be77-4de8dddf91bb","Type":"ContainerDied","Data":"dbdb094c5b2201370ac96031057f837b7a5f471baf615874063d684f5acc5e08"}
May 25 12:16:30.712970 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:30.712935    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cddc9cdc-jkdgr" event={"ID":"d7ea3aea-da4a-49d5-be77-4de8dddf91bb","Type":"ContainerStarted","Data":"422f2a43d9660c0cbe5094e2ff41bd0f03c3a08a2afa3ae679ca0d208a8ffdb5"}
May 25 12:16:56.796233 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:56.796198    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:16:56.798768 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:56.798741    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037bfb47-48b6-4248-9537-b2da35bdf80f-metrics-certs\") pod \"network-metrics-daemon-22jr7\" (UID: \"037bfb47-48b6-4248-9537-b2da35bdf80f\") " pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:16:56.954348 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:56.954307    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nlm6z\""
May 25 12:16:56.961533 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:56.961509    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-22jr7"
May 25 12:16:57.082758 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:57.082675    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-22jr7"]
May 25 12:16:57.086620 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:16:57.086589    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037bfb47_48b6_4248_9537_b2da35bdf80f.slice/crio-f2562a60b53764411f533adb9e7bf75f2bc2777ce1996367c66da5a372b74a1b WatchSource:0}: Error finding container f2562a60b53764411f533adb9e7bf75f2bc2777ce1996367c66da5a372b74a1b: Status 404 returned error can't find the container with id f2562a60b53764411f533adb9e7bf75f2bc2777ce1996367c66da5a372b74a1b
May 25 12:16:57.786142 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:57.786107    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-22jr7" event={"ID":"037bfb47-48b6-4248-9537-b2da35bdf80f","Type":"ContainerStarted","Data":"f2562a60b53764411f533adb9e7bf75f2bc2777ce1996367c66da5a372b74a1b"}
May 25 12:16:58.789865 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:58.789825    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-22jr7" event={"ID":"037bfb47-48b6-4248-9537-b2da35bdf80f","Type":"ContainerStarted","Data":"7944de8f23e9e447e5b6991f983fe4172d215d2dd942504483d73a9c7f506737"}
May 25 12:16:58.789865 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:58.789866    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-22jr7" event={"ID":"037bfb47-48b6-4248-9537-b2da35bdf80f","Type":"ContainerStarted","Data":"507be6cb3746a98a34e1542cbb60a5970e9538f61bcb61275a257212fabc55e8"}
May 25 12:16:58.805625 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:16:58.805575    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-22jr7" podStartSLOduration=252.806627378 podStartE2EDuration="4m13.805561341s" podCreationTimestamp="2026-05-25 12:12:45 +0000 UTC" firstStartedPulling="2026-05-25 12:16:57.088611778 +0000 UTC m=+252.746943167" lastFinishedPulling="2026-05-25 12:16:58.087545747 +0000 UTC m=+253.745877130" observedRunningTime="2026-05-25 12:16:58.804407867 +0000 UTC m=+254.462739273" watchObservedRunningTime="2026-05-25 12:16:58.805561341 +0000 UTC m=+254.463892783"
May 25 12:17:25.500624 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:17:25.500580    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc" podUID="a1ed9528-15dc-44a5-94ce-b457a0eccece"
May 25 12:17:25.862063 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:25.861987    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:17:29.338342 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:29.338289    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:17:29.340674 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:29.340652    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1ed9528-15dc-44a5-94ce-b457a0eccece-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-httbc\" (UID: \"a1ed9528-15dc-44a5-94ce-b457a0eccece\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:17:29.465563 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:29.465528    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-22tjv\""
May 25 12:17:29.472942 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:29.472919    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"
May 25 12:17:29.596397 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:29.596372    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-httbc"]
May 25 12:17:29.598924 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:17:29.598893    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ed9528_15dc_44a5_94ce_b457a0eccece.slice/crio-e51cb018e81770ed0e81e0d17713f577cea4fa99f1230efd02c210a380598fe1 WatchSource:0}: Error finding container e51cb018e81770ed0e81e0d17713f577cea4fa99f1230efd02c210a380598fe1: Status 404 returned error can't find the container with id e51cb018e81770ed0e81e0d17713f577cea4fa99f1230efd02c210a380598fe1
May 25 12:17:29.873908 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:29.873821    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc" event={"ID":"a1ed9528-15dc-44a5-94ce-b457a0eccece","Type":"ContainerStarted","Data":"e51cb018e81770ed0e81e0d17713f577cea4fa99f1230efd02c210a380598fe1"}
May 25 12:17:30.877309 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:30.877252    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc" event={"ID":"a1ed9528-15dc-44a5-94ce-b457a0eccece","Type":"ContainerStarted","Data":"2615fa3a3f2d173867b8c1924707d140470a10954cebb3297bda787b032ad9f5"}
May 25 12:17:30.894128 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:30.894068    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-httbc" podStartSLOduration=277.842793075 podStartE2EDuration="4m38.894050625s" podCreationTimestamp="2026-05-25 12:12:52 +0000 UTC" firstStartedPulling="2026-05-25 12:17:29.600859782 +0000 UTC m=+285.259191166" lastFinishedPulling="2026-05-25 12:17:30.65211733 +0000 UTC m=+286.310448716" observedRunningTime="2026-05-25 12:17:30.893251153 +0000 UTC m=+286.551582559" watchObservedRunningTime="2026-05-25 12:17:30.894050625 +0000 UTC m=+286.552382036"
May 25 12:17:44.851112 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:17:44.851083    2562 kubelet.go:1628] "Image garbage collection succeeded"
May 25 12:20:53.641720 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.641685    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-6xblf"]
May 25 12:20:53.642179 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.641913    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94f30d8-539a-4618-9075-95c2295727d8" containerName="registry"
May 25 12:20:53.642179 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.641923    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94f30d8-539a-4618-9075-95c2295727d8" containerName="registry"
May 25 12:20:53.642179 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.641961    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="a94f30d8-539a-4618-9075-95c2295727d8" containerName="registry"
May 25 12:20:53.643746 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.643730    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:53.646427 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.646405    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 25 12:20:53.646567 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.646450    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-dhbsf\""
May 25 12:20:53.646567 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.646552    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 12:20:53.647694 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.647679    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 12:20:53.647757 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.647727    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 12:20:53.656468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.656447    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-6xblf"]
May 25 12:20:53.676418 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.676390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/749a0cc3-60f8-4cbf-afe2-0511d5e3f684-certificates\") pod \"keda-admission-cf49989db-6xblf\" (UID: \"749a0cc3-60f8-4cbf-afe2-0511d5e3f684\") " pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:53.676549 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.676432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmcgz\" (UniqueName: \"kubernetes.io/projected/749a0cc3-60f8-4cbf-afe2-0511d5e3f684-kube-api-access-pmcgz\") pod \"keda-admission-cf49989db-6xblf\" (UID: \"749a0cc3-60f8-4cbf-afe2-0511d5e3f684\") " pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:53.776787 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.776754    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/749a0cc3-60f8-4cbf-afe2-0511d5e3f684-certificates\") pod \"keda-admission-cf49989db-6xblf\" (UID: \"749a0cc3-60f8-4cbf-afe2-0511d5e3f684\") " pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:53.776916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.776810    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pmcgz\" (UniqueName: \"kubernetes.io/projected/749a0cc3-60f8-4cbf-afe2-0511d5e3f684-kube-api-access-pmcgz\") pod \"keda-admission-cf49989db-6xblf\" (UID: \"749a0cc3-60f8-4cbf-afe2-0511d5e3f684\") " pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:53.776916 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:20:53.776905    2562 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
May 25 12:20:53.776988 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:20:53.776931    2562 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-6xblf: secret "keda-admission-webhooks-certs" not found
May 25 12:20:53.776988 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:20:53.776985    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/749a0cc3-60f8-4cbf-afe2-0511d5e3f684-certificates podName:749a0cc3-60f8-4cbf-afe2-0511d5e3f684 nodeName:}" failed. No retries permitted until 2026-05-25 12:20:54.276965224 +0000 UTC m=+489.935296609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/749a0cc3-60f8-4cbf-afe2-0511d5e3f684-certificates") pod "keda-admission-cf49989db-6xblf" (UID: "749a0cc3-60f8-4cbf-afe2-0511d5e3f684") : secret "keda-admission-webhooks-certs" not found
May 25 12:20:53.785227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:53.785199    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmcgz\" (UniqueName: \"kubernetes.io/projected/749a0cc3-60f8-4cbf-afe2-0511d5e3f684-kube-api-access-pmcgz\") pod \"keda-admission-cf49989db-6xblf\" (UID: \"749a0cc3-60f8-4cbf-afe2-0511d5e3f684\") " pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:54.280176 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:54.280139    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/749a0cc3-60f8-4cbf-afe2-0511d5e3f684-certificates\") pod \"keda-admission-cf49989db-6xblf\" (UID: \"749a0cc3-60f8-4cbf-afe2-0511d5e3f684\") " pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:54.282558 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:54.282538    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/749a0cc3-60f8-4cbf-afe2-0511d5e3f684-certificates\") pod \"keda-admission-cf49989db-6xblf\" (UID: \"749a0cc3-60f8-4cbf-afe2-0511d5e3f684\") " pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:54.553396 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:54.553297    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:54.676903 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:54.676868    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-6xblf"]
May 25 12:20:54.680128 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:20:54.680090    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749a0cc3_60f8_4cbf_afe2_0511d5e3f684.slice/crio-ce1ff89799befe2f6c0dd05ce72a23bf7f4bc1a4236bd6c0d8f66937e310c98c WatchSource:0}: Error finding container ce1ff89799befe2f6c0dd05ce72a23bf7f4bc1a4236bd6c0d8f66937e310c98c: Status 404 returned error can't find the container with id ce1ff89799befe2f6c0dd05ce72a23bf7f4bc1a4236bd6c0d8f66937e310c98c
May 25 12:20:54.683968 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:54.683948    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:20:55.390120 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:55.390082    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-6xblf" event={"ID":"749a0cc3-60f8-4cbf-afe2-0511d5e3f684","Type":"ContainerStarted","Data":"ce1ff89799befe2f6c0dd05ce72a23bf7f4bc1a4236bd6c0d8f66937e310c98c"}
May 25 12:20:58.402164 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:58.402129    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-6xblf" event={"ID":"749a0cc3-60f8-4cbf-afe2-0511d5e3f684","Type":"ContainerStarted","Data":"45fcc739e6a6362f193085752afb4991a4ae0cc1aa30020c6b19f62fd2c3f4ca"}
May 25 12:20:58.402569 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:58.402310    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:20:58.417977 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:20:58.417926    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-6xblf" podStartSLOduration=2.410044691 podStartE2EDuration="5.41791128s" podCreationTimestamp="2026-05-25 12:20:53 +0000 UTC" firstStartedPulling="2026-05-25 12:20:54.684090177 +0000 UTC m=+490.342421559" lastFinishedPulling="2026-05-25 12:20:57.691956759 +0000 UTC m=+493.350288148" observedRunningTime="2026-05-25 12:20:58.41691301 +0000 UTC m=+494.075244416" watchObservedRunningTime="2026-05-25 12:20:58.41791128 +0000 UTC m=+494.076242684"
May 25 12:21:19.406686 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:21:19.406653    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-6xblf"
May 25 12:22:00.363205 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.363121    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"]
May 25 12:22:00.365449 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.365423    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:00.368054 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.368031    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 25 12:22:00.368195 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.368068    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-98x2k\""
May 25 12:22:00.368195 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.368110    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 12:22:00.369283 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.369249    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 12:22:00.376900 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.376870    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"]
May 25 12:22:00.440061 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.440017    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gbs\" (UniqueName: \"kubernetes.io/projected/8d7a3f82-6703-458d-a7eb-29f516883400-kube-api-access-22gbs\") pod \"llmisvc-controller-manager-6b94ff949c-ngfp8\" (UID: \"8d7a3f82-6703-458d-a7eb-29f516883400\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:00.440246 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.440071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d7a3f82-6703-458d-a7eb-29f516883400-cert\") pod \"llmisvc-controller-manager-6b94ff949c-ngfp8\" (UID: \"8d7a3f82-6703-458d-a7eb-29f516883400\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:00.540975 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.540941    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-22gbs\" (UniqueName: \"kubernetes.io/projected/8d7a3f82-6703-458d-a7eb-29f516883400-kube-api-access-22gbs\") pod \"llmisvc-controller-manager-6b94ff949c-ngfp8\" (UID: \"8d7a3f82-6703-458d-a7eb-29f516883400\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:00.541146 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.540995    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d7a3f82-6703-458d-a7eb-29f516883400-cert\") pod \"llmisvc-controller-manager-6b94ff949c-ngfp8\" (UID: \"8d7a3f82-6703-458d-a7eb-29f516883400\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:00.541146 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:22:00.541116    2562 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
May 25 12:22:00.541275 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:22:00.541187    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d7a3f82-6703-458d-a7eb-29f516883400-cert podName:8d7a3f82-6703-458d-a7eb-29f516883400 nodeName:}" failed. No retries permitted until 2026-05-25 12:22:01.041163348 +0000 UTC m=+556.699494732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8d7a3f82-6703-458d-a7eb-29f516883400-cert") pod "llmisvc-controller-manager-6b94ff949c-ngfp8" (UID: "8d7a3f82-6703-458d-a7eb-29f516883400") : secret "llmisvc-webhook-server-cert" not found
May 25 12:22:00.555766 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:00.555733    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gbs\" (UniqueName: \"kubernetes.io/projected/8d7a3f82-6703-458d-a7eb-29f516883400-kube-api-access-22gbs\") pod \"llmisvc-controller-manager-6b94ff949c-ngfp8\" (UID: \"8d7a3f82-6703-458d-a7eb-29f516883400\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:01.045543 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:01.045500    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d7a3f82-6703-458d-a7eb-29f516883400-cert\") pod \"llmisvc-controller-manager-6b94ff949c-ngfp8\" (UID: \"8d7a3f82-6703-458d-a7eb-29f516883400\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:01.048094 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:01.048051    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d7a3f82-6703-458d-a7eb-29f516883400-cert\") pod \"llmisvc-controller-manager-6b94ff949c-ngfp8\" (UID: \"8d7a3f82-6703-458d-a7eb-29f516883400\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:01.276096 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:01.276055    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:01.405515 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:01.405480    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"]
May 25 12:22:01.408825 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:22:01.408799    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8d7a3f82_6703_458d_a7eb_29f516883400.slice/crio-3e8917874d05d43ff401b3b2509fa8c97da1f227df1d932f975ab2417dc7f9d8 WatchSource:0}: Error finding container 3e8917874d05d43ff401b3b2509fa8c97da1f227df1d932f975ab2417dc7f9d8: Status 404 returned error can't find the container with id 3e8917874d05d43ff401b3b2509fa8c97da1f227df1d932f975ab2417dc7f9d8
May 25 12:22:01.569071 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:01.569034    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8" event={"ID":"8d7a3f82-6703-458d-a7eb-29f516883400","Type":"ContainerStarted","Data":"3e8917874d05d43ff401b3b2509fa8c97da1f227df1d932f975ab2417dc7f9d8"}
May 25 12:22:04.579007 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:04.578968    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8" event={"ID":"8d7a3f82-6703-458d-a7eb-29f516883400","Type":"ContainerStarted","Data":"e9d7712cc6bead199ec192edc1630d302b68429b1b6d04491a76720fdd0f86c3"}
May 25 12:22:04.579480 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:04.579104    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:22:04.596691 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:04.596609    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8" podStartSLOduration=1.940861644 podStartE2EDuration="4.596585787s" podCreationTimestamp="2026-05-25 12:22:00 +0000 UTC" firstStartedPulling="2026-05-25 12:22:01.40993099 +0000 UTC m=+557.068262375" lastFinishedPulling="2026-05-25 12:22:04.065655117 +0000 UTC m=+559.723986518" observedRunningTime="2026-05-25 12:22:04.594117597 +0000 UTC m=+560.252449002" watchObservedRunningTime="2026-05-25 12:22:04.596585787 +0000 UTC m=+560.254917193"
May 25 12:22:35.583982 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:22:35.583951    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-ngfp8"
May 25 12:23:11.200468 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.200393    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-qv9q8"]
May 25 12:23:11.202414 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.202398    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:23:11.205128 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.205106    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-d2kmb\""
May 25 12:23:11.205293 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.205115    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 25 12:23:11.214665 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.214639    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-qv9q8"]
May 25 12:23:11.252051 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.252021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h92wg\" (UniqueName: \"kubernetes.io/projected/ba9fee2f-f1aa-4162-af7a-df874a53ba9b-kube-api-access-h92wg\") pod \"odh-model-controller-696fc77849-qv9q8\" (UID: \"ba9fee2f-f1aa-4162-af7a-df874a53ba9b\") " pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:23:11.252051 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.252055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba9fee2f-f1aa-4162-af7a-df874a53ba9b-cert\") pod \"odh-model-controller-696fc77849-qv9q8\" (UID: \"ba9fee2f-f1aa-4162-af7a-df874a53ba9b\") " pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:23:11.353072 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.353033    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h92wg\" (UniqueName: \"kubernetes.io/projected/ba9fee2f-f1aa-4162-af7a-df874a53ba9b-kube-api-access-h92wg\") pod \"odh-model-controller-696fc77849-qv9q8\" (UID: \"ba9fee2f-f1aa-4162-af7a-df874a53ba9b\") " pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:23:11.353072 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.353076    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba9fee2f-f1aa-4162-af7a-df874a53ba9b-cert\") pod \"odh-model-controller-696fc77849-qv9q8\" (UID: \"ba9fee2f-f1aa-4162-af7a-df874a53ba9b\") " pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:23:11.355539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.355512    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba9fee2f-f1aa-4162-af7a-df874a53ba9b-cert\") pod \"odh-model-controller-696fc77849-qv9q8\" (UID: \"ba9fee2f-f1aa-4162-af7a-df874a53ba9b\") " pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:23:11.361567 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.361542    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h92wg\" (UniqueName: \"kubernetes.io/projected/ba9fee2f-f1aa-4162-af7a-df874a53ba9b-kube-api-access-h92wg\") pod \"odh-model-controller-696fc77849-qv9q8\" (UID: \"ba9fee2f-f1aa-4162-af7a-df874a53ba9b\") " pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:23:11.512371 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.512321    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:23:11.636440 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.636407    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-qv9q8"]
May 25 12:23:11.639443 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:23:11.639411    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9fee2f_f1aa_4162_af7a_df874a53ba9b.slice/crio-156a0143500d876cf11d42b521bf23b34c540a633044c576c7a475a297402051 WatchSource:0}: Error finding container 156a0143500d876cf11d42b521bf23b34c540a633044c576c7a475a297402051: Status 404 returned error can't find the container with id 156a0143500d876cf11d42b521bf23b34c540a633044c576c7a475a297402051
May 25 12:23:11.758560 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:11.758526    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-qv9q8" event={"ID":"ba9fee2f-f1aa-4162-af7a-df874a53ba9b","Type":"ContainerStarted","Data":"156a0143500d876cf11d42b521bf23b34c540a633044c576c7a475a297402051"}
May 25 12:23:14.767794 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:14.767758    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-qv9q8" event={"ID":"ba9fee2f-f1aa-4162-af7a-df874a53ba9b","Type":"ContainerStarted","Data":"02b1f406c400e6be8448d60f54cb91fa0462ed4ec4b02bcdb40ea0fccea49888"}
May 25 12:23:14.768302 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:14.767874    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:23:14.783990 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:14.783936    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-qv9q8" podStartSLOduration=1.123756577 podStartE2EDuration="3.783920751s" podCreationTimestamp="2026-05-25 12:23:11 +0000 UTC" firstStartedPulling="2026-05-25 12:23:11.640666712 +0000 UTC m=+627.298998110" lastFinishedPulling="2026-05-25 12:23:14.3008309 +0000 UTC m=+629.959162284" observedRunningTime="2026-05-25 12:23:14.782880756 +0000 UTC m=+630.441212161" watchObservedRunningTime="2026-05-25 12:23:14.783920751 +0000 UTC m=+630.442252157"
May 25 12:23:25.774028 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:23:25.773996    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-qv9q8"
May 25 12:37:48.088769 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.088734    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"]
May 25 12:37:48.091927 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.091909    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.094962 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.094933    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:37:48.095164 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.095144    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
May 25 12:37:48.095474 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.095180    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:37:48.095786 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.095175    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
May 25 12:37:48.096047 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.095791    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 12:37:48.102597 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.102570    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"]
May 25 12:37:48.193246 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.193211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.193246 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.193255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1ce8c5-89c6-444a-9670-7692d18f53d2-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.193465 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.193308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7c64\" (UniqueName: \"kubernetes.io/projected/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kube-api-access-h7c64\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.193465 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.193413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1ce8c5-89c6-444a-9670-7692d18f53d2-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.294710 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.294677    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.294710 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.294719    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1ce8c5-89c6-444a-9670-7692d18f53d2-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.294930 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.294761    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h7c64\" (UniqueName: \"kubernetes.io/projected/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kube-api-access-h7c64\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.294930 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.294797    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1ce8c5-89c6-444a-9670-7692d18f53d2-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.295162 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.295137    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.295497 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.295481    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1ce8c5-89c6-444a-9670-7692d18f53d2-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.297236 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.297217    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1ce8c5-89c6-444a-9670-7692d18f53d2-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.304722 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.304693    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7c64\" (UniqueName: \"kubernetes.io/projected/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kube-api-access-h7c64\") pod \"isvc-paddle-predictor-6b8b7cfb4b-d992h\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.405536 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.405445    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:37:48.531114 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.531081    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"]
May 25 12:37:48.534847 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:37:48.534817    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1ce8c5_89c6_444a_9670_7692d18f53d2.slice/crio-c14b07573a97b4b738d41ed8cc07e4db892f2f376981925479d3177380dda9e4 WatchSource:0}: Error finding container c14b07573a97b4b738d41ed8cc07e4db892f2f376981925479d3177380dda9e4: Status 404 returned error can't find the container with id c14b07573a97b4b738d41ed8cc07e4db892f2f376981925479d3177380dda9e4
May 25 12:37:48.536814 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:48.536797    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:37:49.049671 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:49.049632    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" event={"ID":"6e1ce8c5-89c6-444a-9670-7692d18f53d2","Type":"ContainerStarted","Data":"c14b07573a97b4b738d41ed8cc07e4db892f2f376981925479d3177380dda9e4"}
May 25 12:37:53.064362 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:53.064324    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" event={"ID":"6e1ce8c5-89c6-444a-9670-7692d18f53d2","Type":"ContainerStarted","Data":"561ee619505100e11c59a4bc006714b8ad9b7e4a417cc3653ef73490b2c5ffeb"}
May 25 12:37:58.082251 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:58.082213    2562 generic.go:358] "Generic (PLEG): container finished" podID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerID="561ee619505100e11c59a4bc006714b8ad9b7e4a417cc3653ef73490b2c5ffeb" exitCode=0
May 25 12:37:58.082687 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:37:58.082297    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" event={"ID":"6e1ce8c5-89c6-444a-9670-7692d18f53d2","Type":"ContainerDied","Data":"561ee619505100e11c59a4bc006714b8ad9b7e4a417cc3653ef73490b2c5ffeb"}
May 25 12:38:10.125580 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:10.125538    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" event={"ID":"6e1ce8c5-89c6-444a-9670-7692d18f53d2","Type":"ContainerStarted","Data":"97dde2fa4e2157c68e89fed99490d83b2a4d26ad1b2d1f51dd539009db36d315"}
May 25 12:38:12.133441 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:12.133407    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" event={"ID":"6e1ce8c5-89c6-444a-9670-7692d18f53d2","Type":"ContainerStarted","Data":"4edbc8dca6f1c93cf517a08ad9aa086f83141070a02af12a601da9edc474202a"}
May 25 12:38:12.133859 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:12.133603    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:38:12.152725 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:12.152672    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" podStartSLOduration=0.965223313 podStartE2EDuration="24.152658841s" podCreationTimestamp="2026-05-25 12:37:48 +0000 UTC" firstStartedPulling="2026-05-25 12:37:48.536919768 +0000 UTC m=+1504.195251150" lastFinishedPulling="2026-05-25 12:38:11.724355296 +0000 UTC m=+1527.382686678" observedRunningTime="2026-05-25 12:38:12.151274841 +0000 UTC m=+1527.809606247" watchObservedRunningTime="2026-05-25 12:38:12.152658841 +0000 UTC m=+1527.810990269"
May 25 12:38:13.136133 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:13.136100    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:38:13.137254 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:13.137227    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:38:14.138953 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:14.138911    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:38:19.143245 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:19.143214    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:38:19.143859 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:19.143830    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:38:29.144422 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:29.144379    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:38:39.144639 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:39.144599    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:38:49.144157 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:49.144114    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:38:59.145224 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:38:59.145191    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:39:09.567174 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.567139    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"]
May 25 12:39:09.567772 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.567620    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container" containerID="cri-o://97dde2fa4e2157c68e89fed99490d83b2a4d26ad1b2d1f51dd539009db36d315" gracePeriod=30
May 25 12:39:09.567772 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.567667    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kube-rbac-proxy" containerID="cri-o://4edbc8dca6f1c93cf517a08ad9aa086f83141070a02af12a601da9edc474202a" gracePeriod=30
May 25 12:39:09.664744 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.664701    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"]
May 25 12:39:09.669352 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.669319    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.671994 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.671965    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
May 25 12:39:09.671994 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.671970    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
May 25 12:39:09.676869 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.676844    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"]
May 25 12:39:09.709286 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.709233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.709443 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.709328    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70203b85-9bf3-45f0-9472-c26ebfbb8f26-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.709443 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.709359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6rd\" (UniqueName: \"kubernetes.io/projected/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kube-api-access-ln6rd\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.709443 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.709391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70203b85-9bf3-45f0-9472-c26ebfbb8f26-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.810120 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.810084    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70203b85-9bf3-45f0-9472-c26ebfbb8f26-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.810120 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.810127    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6rd\" (UniqueName: \"kubernetes.io/projected/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kube-api-access-ln6rd\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.810408 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.810150    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70203b85-9bf3-45f0-9472-c26ebfbb8f26-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.810408 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.810193    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.810408 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:39:09.810339    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
May 25 12:39:09.810408 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:39:09.810394    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70203b85-9bf3-45f0-9472-c26ebfbb8f26-proxy-tls podName:70203b85-9bf3-45f0-9472-c26ebfbb8f26 nodeName:}" failed. No retries permitted until 2026-05-25 12:39:10.310376543 +0000 UTC m=+1585.968707926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/70203b85-9bf3-45f0-9472-c26ebfbb8f26-proxy-tls") pod "isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" (UID: "70203b85-9bf3-45f0-9472-c26ebfbb8f26") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
May 25 12:39:09.810660 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.810640    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.810804 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.810785    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70203b85-9bf3-45f0-9472-c26ebfbb8f26-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:09.818655 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:09.818586    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6rd\" (UniqueName: \"kubernetes.io/projected/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kube-api-access-ln6rd\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:10.282343 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:10.282309    2562 generic.go:358] "Generic (PLEG): container finished" podID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerID="4edbc8dca6f1c93cf517a08ad9aa086f83141070a02af12a601da9edc474202a" exitCode=2
May 25 12:39:10.282515 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:10.282384    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" event={"ID":"6e1ce8c5-89c6-444a-9670-7692d18f53d2","Type":"ContainerDied","Data":"4edbc8dca6f1c93cf517a08ad9aa086f83141070a02af12a601da9edc474202a"}
May 25 12:39:10.314901 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:10.314868    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70203b85-9bf3-45f0-9472-c26ebfbb8f26-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:10.317449 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:10.317418    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70203b85-9bf3-45f0-9472-c26ebfbb8f26-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:10.581092 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:10.581001    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:10.707019 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:10.706992    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"]
May 25 12:39:10.709316 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:39:10.709287    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70203b85_9bf3_45f0_9472_c26ebfbb8f26.slice/crio-97b507d27bd7fdeb3af941a4dff8437496984df5e5ab6f8b1e7d960e4954e3b3 WatchSource:0}: Error finding container 97b507d27bd7fdeb3af941a4dff8437496984df5e5ab6f8b1e7d960e4954e3b3: Status 404 returned error can't find the container with id 97b507d27bd7fdeb3af941a4dff8437496984df5e5ab6f8b1e7d960e4954e3b3
May 25 12:39:11.286582 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:11.286542    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" event={"ID":"70203b85-9bf3-45f0-9472-c26ebfbb8f26","Type":"ContainerStarted","Data":"2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76"}
May 25 12:39:11.286582 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:11.286581    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" event={"ID":"70203b85-9bf3-45f0-9472-c26ebfbb8f26","Type":"ContainerStarted","Data":"97b507d27bd7fdeb3af941a4dff8437496984df5e5ab6f8b1e7d960e4954e3b3"}
May 25 12:39:12.291076 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.291048    2562 generic.go:358] "Generic (PLEG): container finished" podID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerID="97dde2fa4e2157c68e89fed99490d83b2a4d26ad1b2d1f51dd539009db36d315" exitCode=0
May 25 12:39:12.291458 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.291121    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" event={"ID":"6e1ce8c5-89c6-444a-9670-7692d18f53d2","Type":"ContainerDied","Data":"97dde2fa4e2157c68e89fed99490d83b2a4d26ad1b2d1f51dd539009db36d315"}
May 25 12:39:12.415529 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.415506    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:39:12.532103 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.532003    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kserve-provision-location\") pod \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") "
May 25 12:39:12.532103 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.532049    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7c64\" (UniqueName: \"kubernetes.io/projected/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kube-api-access-h7c64\") pod \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") "
May 25 12:39:12.532373 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.532110    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1ce8c5-89c6-444a-9670-7692d18f53d2-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") "
May 25 12:39:12.532373 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.532137    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1ce8c5-89c6-444a-9670-7692d18f53d2-proxy-tls\") pod \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\" (UID: \"6e1ce8c5-89c6-444a-9670-7692d18f53d2\") "
May 25 12:39:12.532465 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.532435    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1ce8c5-89c6-444a-9670-7692d18f53d2-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "6e1ce8c5-89c6-444a-9670-7692d18f53d2" (UID: "6e1ce8c5-89c6-444a-9670-7692d18f53d2"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:39:12.534327 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.534300    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kube-api-access-h7c64" (OuterVolumeSpecName: "kube-api-access-h7c64") pod "6e1ce8c5-89c6-444a-9670-7692d18f53d2" (UID: "6e1ce8c5-89c6-444a-9670-7692d18f53d2"). InnerVolumeSpecName "kube-api-access-h7c64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:39:12.534443 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.534398    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1ce8c5-89c6-444a-9670-7692d18f53d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e1ce8c5-89c6-444a-9670-7692d18f53d2" (UID: "6e1ce8c5-89c6-444a-9670-7692d18f53d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:39:12.540897 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.540873    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e1ce8c5-89c6-444a-9670-7692d18f53d2" (UID: "6e1ce8c5-89c6-444a-9670-7692d18f53d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:39:12.632681 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.632635    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:39:12.632681 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.632673    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7c64\" (UniqueName: \"kubernetes.io/projected/6e1ce8c5-89c6-444a-9670-7692d18f53d2-kube-api-access-h7c64\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:39:12.632681 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.632688    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1ce8c5-89c6-444a-9670-7692d18f53d2-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:39:12.632909 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:12.632705    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1ce8c5-89c6-444a-9670-7692d18f53d2-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:39:13.295435 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:13.295400    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h" event={"ID":"6e1ce8c5-89c6-444a-9670-7692d18f53d2","Type":"ContainerDied","Data":"c14b07573a97b4b738d41ed8cc07e4db892f2f376981925479d3177380dda9e4"}
May 25 12:39:13.295916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:13.295449    2562 scope.go:117] "RemoveContainer" containerID="4edbc8dca6f1c93cf517a08ad9aa086f83141070a02af12a601da9edc474202a"
May 25 12:39:13.295916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:13.295459    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"
May 25 12:39:13.303008 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:13.302988    2562 scope.go:117] "RemoveContainer" containerID="97dde2fa4e2157c68e89fed99490d83b2a4d26ad1b2d1f51dd539009db36d315"
May 25 12:39:13.310196 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:13.310178    2562 scope.go:117] "RemoveContainer" containerID="561ee619505100e11c59a4bc006714b8ad9b7e4a417cc3653ef73490b2c5ffeb"
May 25 12:39:13.314069 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:13.314040    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"]
May 25 12:39:13.316938 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:13.316917    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-d992h"]
May 25 12:39:14.958600 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:14.958567    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" path="/var/lib/kubelet/pods/6e1ce8c5-89c6-444a-9670-7692d18f53d2/volumes"
May 25 12:39:15.303557 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:15.303525    2562 generic.go:358] "Generic (PLEG): container finished" podID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerID="2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76" exitCode=0
May 25 12:39:15.303714 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:15.303600    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" event={"ID":"70203b85-9bf3-45f0-9472-c26ebfbb8f26","Type":"ContainerDied","Data":"2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76"}
May 25 12:39:16.308335 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:16.308299    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" event={"ID":"70203b85-9bf3-45f0-9472-c26ebfbb8f26","Type":"ContainerStarted","Data":"5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e"}
May 25 12:39:16.308335 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:16.308339    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" event={"ID":"70203b85-9bf3-45f0-9472-c26ebfbb8f26","Type":"ContainerStarted","Data":"871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378"}
May 25 12:39:16.308814 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:16.308629    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:16.308814 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:16.308752    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:16.310092 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:16.310065    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 25 12:39:16.325460 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:16.325398    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podStartSLOduration=7.325379891 podStartE2EDuration="7.325379891s" podCreationTimestamp="2026-05-25 12:39:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:39:16.324564002 +0000 UTC m=+1591.982895441" watchObservedRunningTime="2026-05-25 12:39:16.325379891 +0000 UTC m=+1591.983711297"
May 25 12:39:17.310826 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:17.310792    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 25 12:39:22.315454 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:22.315423    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:39:22.316043 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:22.316017    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 25 12:39:32.316944 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:32.316894    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 25 12:39:42.317050 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:42.316957    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 25 12:39:52.316008 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:39:52.315964    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 25 12:40:02.317216 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:02.317184    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:40:11.168591 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.168549    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"]
May 25 12:40:11.169064 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.168972    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container" containerID="cri-o://871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378" gracePeriod=30
May 25 12:40:11.169064 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.169012    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kube-rbac-proxy" containerID="cri-o://5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e" gracePeriod=30
May 25 12:40:11.265572 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.265536    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"]
May 25 12:40:11.265828 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.265815    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="storage-initializer"
May 25 12:40:11.265876 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.265829    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="storage-initializer"
May 25 12:40:11.265876 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.265839    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container"
May 25 12:40:11.265876 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.265845    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container"
May 25 12:40:11.265876 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.265858    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kube-rbac-proxy"
May 25 12:40:11.265876 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.265863    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kube-rbac-proxy"
May 25 12:40:11.266015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.265909    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kserve-container"
May 25 12:40:11.266015 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.265918    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="6e1ce8c5-89c6-444a-9670-7692d18f53d2" containerName="kube-rbac-proxy"
May 25 12:40:11.270006 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.269982    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.272504 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.272476    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
May 25 12:40:11.272632 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.272479    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
May 25 12:40:11.274553 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.274528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88p55\" (UniqueName: \"kubernetes.io/projected/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kube-api-access-88p55\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.274689 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.274561    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ea7dc2-a634-4d88-a42d-26cce3db800b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.274689 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.274588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.274806 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.274696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ea7dc2-a634-4d88-a42d-26cce3db800b-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.278633 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.278577    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"]
May 25 12:40:11.375593 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.375560    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-88p55\" (UniqueName: \"kubernetes.io/projected/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kube-api-access-88p55\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.375805 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.375601    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ea7dc2-a634-4d88-a42d-26cce3db800b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.375805 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.375636    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.375805 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.375684    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ea7dc2-a634-4d88-a42d-26cce3db800b-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.376078 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.376060    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.376376 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.376356    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ea7dc2-a634-4d88-a42d-26cce3db800b-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.378179 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.378160    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ea7dc2-a634-4d88-a42d-26cce3db800b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.383337 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.383313    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88p55\" (UniqueName: \"kubernetes.io/projected/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kube-api-access-88p55\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.464296 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.464188    2562 generic.go:358] "Generic (PLEG): container finished" podID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerID="5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e" exitCode=2
May 25 12:40:11.464296 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.464243    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" event={"ID":"70203b85-9bf3-45f0-9472-c26ebfbb8f26","Type":"ContainerDied","Data":"5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e"}
May 25 12:40:11.583047 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.583013    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:11.708851 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:11.708704    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"]
May 25 12:40:11.711439 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:40:11.711408    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ea7dc2_a634_4d88_a42d_26cce3db800b.slice/crio-ef132e9a384a3f7080a657ddf52ff9ecb66967a35694e1b5c8d2b53d0140a806 WatchSource:0}: Error finding container ef132e9a384a3f7080a657ddf52ff9ecb66967a35694e1b5c8d2b53d0140a806: Status 404 returned error can't find the container with id ef132e9a384a3f7080a657ddf52ff9ecb66967a35694e1b5c8d2b53d0140a806
May 25 12:40:12.311226 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:12.311177    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
May 25 12:40:12.316399 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:12.316364    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 25 12:40:12.468145 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:12.468112    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" event={"ID":"a0ea7dc2-a634-4d88-a42d-26cce3db800b","Type":"ContainerStarted","Data":"58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462"}
May 25 12:40:12.468145 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:12.468147    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" event={"ID":"a0ea7dc2-a634-4d88-a42d-26cce3db800b","Type":"ContainerStarted","Data":"ef132e9a384a3f7080a657ddf52ff9ecb66967a35694e1b5c8d2b53d0140a806"}
May 25 12:40:14.019756 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.019730    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:40:14.095011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.094914    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70203b85-9bf3-45f0-9472-c26ebfbb8f26-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") "
May 25 12:40:14.095011 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.094976    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln6rd\" (UniqueName: \"kubernetes.io/projected/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kube-api-access-ln6rd\") pod \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") "
May 25 12:40:14.095245 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.095027    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kserve-provision-location\") pod \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") "
May 25 12:40:14.095245 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.095076    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70203b85-9bf3-45f0-9472-c26ebfbb8f26-proxy-tls\") pod \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\" (UID: \"70203b85-9bf3-45f0-9472-c26ebfbb8f26\") "
May 25 12:40:14.095377 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.095344    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70203b85-9bf3-45f0-9472-c26ebfbb8f26-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "70203b85-9bf3-45f0-9472-c26ebfbb8f26" (UID: "70203b85-9bf3-45f0-9472-c26ebfbb8f26"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:40:14.097174 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.097137    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kube-api-access-ln6rd" (OuterVolumeSpecName: "kube-api-access-ln6rd") pod "70203b85-9bf3-45f0-9472-c26ebfbb8f26" (UID: "70203b85-9bf3-45f0-9472-c26ebfbb8f26"). InnerVolumeSpecName "kube-api-access-ln6rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:40:14.097303 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.097210    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70203b85-9bf3-45f0-9472-c26ebfbb8f26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70203b85-9bf3-45f0-9472-c26ebfbb8f26" (UID: "70203b85-9bf3-45f0-9472-c26ebfbb8f26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:40:14.104480 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.104452    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70203b85-9bf3-45f0-9472-c26ebfbb8f26" (UID: "70203b85-9bf3-45f0-9472-c26ebfbb8f26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:40:14.195961 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.195920    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:40:14.195961 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.195958    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70203b85-9bf3-45f0-9472-c26ebfbb8f26-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:40:14.196173 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.195975    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70203b85-9bf3-45f0-9472-c26ebfbb8f26-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:40:14.196173 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.195988    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln6rd\" (UniqueName: \"kubernetes.io/projected/70203b85-9bf3-45f0-9472-c26ebfbb8f26-kube-api-access-ln6rd\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:40:14.475854 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.475812    2562 generic.go:358] "Generic (PLEG): container finished" podID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerID="871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378" exitCode=0
May 25 12:40:14.476040 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.475900    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"
May 25 12:40:14.476040 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.475895    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" event={"ID":"70203b85-9bf3-45f0-9472-c26ebfbb8f26","Type":"ContainerDied","Data":"871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378"}
May 25 12:40:14.476040 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.475944    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc" event={"ID":"70203b85-9bf3-45f0-9472-c26ebfbb8f26","Type":"ContainerDied","Data":"97b507d27bd7fdeb3af941a4dff8437496984df5e5ab6f8b1e7d960e4954e3b3"}
May 25 12:40:14.476040 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.475965    2562 scope.go:117] "RemoveContainer" containerID="5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e"
May 25 12:40:14.484598 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.484577    2562 scope.go:117] "RemoveContainer" containerID="871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378"
May 25 12:40:14.491876 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.491858    2562 scope.go:117] "RemoveContainer" containerID="2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76"
May 25 12:40:14.497235 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.497211    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"]
May 25 12:40:14.499422 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.499400    2562 scope.go:117] "RemoveContainer" containerID="5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e"
May 25 12:40:14.499695 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:40:14.499677    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e\": container with ID starting with 5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e not found: ID does not exist" containerID="5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e"
May 25 12:40:14.499752 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.499705    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e"} err="failed to get container status \"5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e\": rpc error: code = NotFound desc = could not find container \"5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e\": container with ID starting with 5cd9482e62450c5088f252ff7e3a9bc4a6c1864cd9ec53cb50d33001804b2b1e not found: ID does not exist"
May 25 12:40:14.499752 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.499736    2562 scope.go:117] "RemoveContainer" containerID="871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378"
May 25 12:40:14.499987 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:40:14.499970    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378\": container with ID starting with 871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378 not found: ID does not exist" containerID="871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378"
May 25 12:40:14.500027 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.499994    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378"} err="failed to get container status \"871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378\": rpc error: code = NotFound desc = could not find container \"871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378\": container with ID starting with 871c1f704bd130d3a8062ae32a2688c9f134ad310c9ace53e97eb171c2860378 not found: ID does not exist"
May 25 12:40:14.500027 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.500015    2562 scope.go:117] "RemoveContainer" containerID="2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76"
May 25 12:40:14.500294 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:40:14.500270    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76\": container with ID starting with 2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76 not found: ID does not exist" containerID="2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76"
May 25 12:40:14.500496 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.500297    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76"} err="failed to get container status \"2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76\": rpc error: code = NotFound desc = could not find container \"2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76\": container with ID starting with 2cb4ab4db48704152761f21abd0e862772a4fead99107eb3b49a21343fe52b76 not found: ID does not exist"
May 25 12:40:14.502097 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.502073    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-dflgc"]
May 25 12:40:14.954452 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:14.954371    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" path="/var/lib/kubelet/pods/70203b85-9bf3-45f0-9472-c26ebfbb8f26/volumes"
May 25 12:40:16.484106 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:16.484069    2562 generic.go:358] "Generic (PLEG): container finished" podID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerID="58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462" exitCode=0
May 25 12:40:16.484499 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:16.484141    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" event={"ID":"a0ea7dc2-a634-4d88-a42d-26cce3db800b","Type":"ContainerDied","Data":"58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462"}
May 25 12:40:17.488860 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:17.488823    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" event={"ID":"a0ea7dc2-a634-4d88-a42d-26cce3db800b","Type":"ContainerStarted","Data":"cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44"}
May 25 12:40:17.488860 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:17.488865    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" event={"ID":"a0ea7dc2-a634-4d88-a42d-26cce3db800b","Type":"ContainerStarted","Data":"eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8"}
May 25 12:40:17.489422 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:17.489091    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:17.507688 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:17.507640    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podStartSLOduration=6.507626568 podStartE2EDuration="6.507626568s" podCreationTimestamp="2026-05-25 12:40:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:40:17.506173423 +0000 UTC m=+1653.164504829" watchObservedRunningTime="2026-05-25 12:40:17.507626568 +0000 UTC m=+1653.165957972"
May 25 12:40:18.492127 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:18.492087    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:18.493327 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:18.493300    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 12:40:19.494779 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:19.494740    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 12:40:24.499559 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:24.499528    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:40:24.500131 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:24.500101    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 12:40:34.500804 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:34.500759    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 12:40:44.500156 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:44.500113    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 12:40:54.501081 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:40:54.501040    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 12:41:04.500545 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:04.500460    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:41:13.155285 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.155237    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"]
May 25 12:41:13.155685 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.155568    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container" containerID="cri-o://eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8" gracePeriod=30
May 25 12:41:13.155685 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.155599    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kube-rbac-proxy" containerID="cri-o://cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44" gracePeriod=30
May 25 12:41:13.251738 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.251704    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"]
May 25 12:41:13.251978 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.251965    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="storage-initializer"
May 25 12:41:13.252027 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.251980    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="storage-initializer"
May 25 12:41:13.252027 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.252015    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container"
May 25 12:41:13.252027 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.252021    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container"
May 25 12:41:13.252118 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.252028    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kube-rbac-proxy"
May 25 12:41:13.252118 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.252034    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kube-rbac-proxy"
May 25 12:41:13.252118 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.252077    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kube-rbac-proxy"
May 25 12:41:13.252118 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.252084    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="70203b85-9bf3-45f0-9472-c26ebfbb8f26" containerName="kserve-container"
May 25 12:41:13.256153 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.256134    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.258449 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.258425    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
May 25 12:41:13.258569 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.258486    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
May 25 12:41:13.265413 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.265385    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"]
May 25 12:41:13.309171 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.309135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94fc6618-bf69-4a33-8784-9a084a8f8dee-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.309389 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.309176    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94fc6618-bf69-4a33-8784-9a084a8f8dee-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.309389 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.309322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94fc6618-bf69-4a33-8784-9a084a8f8dee-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.309389 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.309361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h25x4\" (UniqueName: \"kubernetes.io/projected/94fc6618-bf69-4a33-8784-9a084a8f8dee-kube-api-access-h25x4\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.410529 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.410414    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94fc6618-bf69-4a33-8784-9a084a8f8dee-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.410529 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.410474    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94fc6618-bf69-4a33-8784-9a084a8f8dee-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.410761 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.410540    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94fc6618-bf69-4a33-8784-9a084a8f8dee-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.410761 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.410567    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h25x4\" (UniqueName: \"kubernetes.io/projected/94fc6618-bf69-4a33-8784-9a084a8f8dee-kube-api-access-h25x4\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.410761 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:41:13.410700    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
May 25 12:41:13.410910 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:41:13.410778    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fc6618-bf69-4a33-8784-9a084a8f8dee-proxy-tls podName:94fc6618-bf69-4a33-8784-9a084a8f8dee nodeName:}" failed. No retries permitted until 2026-05-25 12:41:13.910761142 +0000 UTC m=+1709.569092530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/94fc6618-bf69-4a33-8784-9a084a8f8dee-proxy-tls") pod "isvc-pmml-predictor-8bb578669-h7l8g" (UID: "94fc6618-bf69-4a33-8784-9a084a8f8dee") : secret "isvc-pmml-predictor-serving-cert" not found
May 25 12:41:13.410972 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.410922    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94fc6618-bf69-4a33-8784-9a084a8f8dee-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.411163 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.411141    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94fc6618-bf69-4a33-8784-9a084a8f8dee-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.419241 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.419213    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h25x4\" (UniqueName: \"kubernetes.io/projected/94fc6618-bf69-4a33-8784-9a084a8f8dee-kube-api-access-h25x4\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.643244 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.643210    2562 generic.go:358] "Generic (PLEG): container finished" podID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerID="cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44" exitCode=2
May 25 12:41:13.643427 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.643275    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" event={"ID":"a0ea7dc2-a634-4d88-a42d-26cce3db800b","Type":"ContainerDied","Data":"cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44"}
May 25 12:41:13.914768 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.914733    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94fc6618-bf69-4a33-8784-9a084a8f8dee-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:13.917302 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:13.917246    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94fc6618-bf69-4a33-8784-9a084a8f8dee-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-h7l8g\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:14.167593 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:14.167494    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:14.290731 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:14.290568    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"]
May 25 12:41:14.293289 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:41:14.293246    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fc6618_bf69_4a33_8784_9a084a8f8dee.slice/crio-c4c2617363ce90d1d4a939ad284497f39bd64bf4511964c7801cfa642faf403c WatchSource:0}: Error finding container c4c2617363ce90d1d4a939ad284497f39bd64bf4511964c7801cfa642faf403c: Status 404 returned error can't find the container with id c4c2617363ce90d1d4a939ad284497f39bd64bf4511964c7801cfa642faf403c
May 25 12:41:14.495222 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:14.495179    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 25 12:41:14.500073 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:14.500039    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 12:41:14.651198 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:14.651162    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" event={"ID":"94fc6618-bf69-4a33-8784-9a084a8f8dee","Type":"ContainerStarted","Data":"819cc911107f018f8a0c571bffbd8cb8d8d92c9567baa5a4edbd05174c16d8e6"}
May 25 12:41:14.651198 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:14.651202    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" event={"ID":"94fc6618-bf69-4a33-8784-9a084a8f8dee","Type":"ContainerStarted","Data":"c4c2617363ce90d1d4a939ad284497f39bd64bf4511964c7801cfa642faf403c"}
May 25 12:41:15.900114 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:15.900090    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:41:16.031442 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.031344    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ea7dc2-a634-4d88-a42d-26cce3db800b-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") "
May 25 12:41:16.031442 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.031423    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kserve-provision-location\") pod \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") "
May 25 12:41:16.031442 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.031442    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ea7dc2-a634-4d88-a42d-26cce3db800b-proxy-tls\") pod \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") "
May 25 12:41:16.031744 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.031464    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88p55\" (UniqueName: \"kubernetes.io/projected/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kube-api-access-88p55\") pod \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\" (UID: \"a0ea7dc2-a634-4d88-a42d-26cce3db800b\") "
May 25 12:41:16.031744 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.031716    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ea7dc2-a634-4d88-a42d-26cce3db800b-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "a0ea7dc2-a634-4d88-a42d-26cce3db800b" (UID: "a0ea7dc2-a634-4d88-a42d-26cce3db800b"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:41:16.033567 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.033533    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ea7dc2-a634-4d88-a42d-26cce3db800b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0ea7dc2-a634-4d88-a42d-26cce3db800b" (UID: "a0ea7dc2-a634-4d88-a42d-26cce3db800b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:41:16.033683 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.033572    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kube-api-access-88p55" (OuterVolumeSpecName: "kube-api-access-88p55") pod "a0ea7dc2-a634-4d88-a42d-26cce3db800b" (UID: "a0ea7dc2-a634-4d88-a42d-26cce3db800b"). InnerVolumeSpecName "kube-api-access-88p55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:41:16.041129 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.041098    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0ea7dc2-a634-4d88-a42d-26cce3db800b" (UID: "a0ea7dc2-a634-4d88-a42d-26cce3db800b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:41:16.132699 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.132664    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:41:16.132699 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.132697    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ea7dc2-a634-4d88-a42d-26cce3db800b-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:41:16.132874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.132710    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88p55\" (UniqueName: \"kubernetes.io/projected/a0ea7dc2-a634-4d88-a42d-26cce3db800b-kube-api-access-88p55\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:41:16.132874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.132720    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ea7dc2-a634-4d88-a42d-26cce3db800b-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:41:16.658374 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.658341    2562 generic.go:358] "Generic (PLEG): container finished" podID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerID="eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8" exitCode=0
May 25 12:41:16.658539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.658419    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" event={"ID":"a0ea7dc2-a634-4d88-a42d-26cce3db800b","Type":"ContainerDied","Data":"eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8"}
May 25 12:41:16.658539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.658457    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx" event={"ID":"a0ea7dc2-a634-4d88-a42d-26cce3db800b","Type":"ContainerDied","Data":"ef132e9a384a3f7080a657ddf52ff9ecb66967a35694e1b5c8d2b53d0140a806"}
May 25 12:41:16.658539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.658463    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"
May 25 12:41:16.658539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.658473    2562 scope.go:117] "RemoveContainer" containerID="cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44"
May 25 12:41:16.666956 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.666935    2562 scope.go:117] "RemoveContainer" containerID="eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8"
May 25 12:41:16.673678 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.673661    2562 scope.go:117] "RemoveContainer" containerID="58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462"
May 25 12:41:16.679745 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.679723    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"]
May 25 12:41:16.681010 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.680998    2562 scope.go:117] "RemoveContainer" containerID="cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44"
May 25 12:41:16.681283 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:41:16.681247    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44\": container with ID starting with cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44 not found: ID does not exist" containerID="cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44"
May 25 12:41:16.681368 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.681295    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44"} err="failed to get container status \"cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44\": rpc error: code = NotFound desc = could not find container \"cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44\": container with ID starting with cd7314c245016244dfe670f2ed5504cbfb72a7334c2feb0480e04febc0585d44 not found: ID does not exist"
May 25 12:41:16.681368 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.681319    2562 scope.go:117] "RemoveContainer" containerID="eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8"
May 25 12:41:16.681655 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:41:16.681598    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8\": container with ID starting with eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8 not found: ID does not exist" containerID="eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8"
May 25 12:41:16.681738 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.681664    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8"} err="failed to get container status \"eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8\": rpc error: code = NotFound desc = could not find container \"eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8\": container with ID starting with eb647413a7c69ef3faa0da2c199fe3ee5d71e53bb8069283329b8d6d862ec9a8 not found: ID does not exist"
May 25 12:41:16.681738 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.681687    2562 scope.go:117] "RemoveContainer" containerID="58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462"
May 25 12:41:16.682032 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:41:16.682006    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462\": container with ID starting with 58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462 not found: ID does not exist" containerID="58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462"
May 25 12:41:16.682099 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.682040    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462"} err="failed to get container status \"58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462\": rpc error: code = NotFound desc = could not find container \"58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462\": container with ID starting with 58672e7970a4d76e981cb8db0ae5a506eb581cbbba848fa8a95b1f4ca8c57462 not found: ID does not exist"
May 25 12:41:16.683433 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.683413    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-xl6wx"]
May 25 12:41:16.954453 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:16.954374    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" path="/var/lib/kubelet/pods/a0ea7dc2-a634-4d88-a42d-26cce3db800b/volumes"
May 25 12:41:18.666342 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:18.666308    2562 generic.go:358] "Generic (PLEG): container finished" podID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerID="819cc911107f018f8a0c571bffbd8cb8d8d92c9567baa5a4edbd05174c16d8e6" exitCode=0
May 25 12:41:18.666718 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:18.666383    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" event={"ID":"94fc6618-bf69-4a33-8784-9a084a8f8dee","Type":"ContainerDied","Data":"819cc911107f018f8a0c571bffbd8cb8d8d92c9567baa5a4edbd05174c16d8e6"}
May 25 12:41:25.690210 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:25.690183    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" event={"ID":"94fc6618-bf69-4a33-8784-9a084a8f8dee","Type":"ContainerStarted","Data":"05ccb5b944e3f89e2d6afbc84c720df81046cbd3a91906ee5ce5461372648f7f"}
May 25 12:41:26.694244 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:26.694211    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" event={"ID":"94fc6618-bf69-4a33-8784-9a084a8f8dee","Type":"ContainerStarted","Data":"7449349c406c7d11584835185366c3cdeac02915df38043536220bf46b7c4d13"}
May 25 12:41:26.711382 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:26.711334    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podStartSLOduration=6.782663775 podStartE2EDuration="13.711319426s" podCreationTimestamp="2026-05-25 12:41:13 +0000 UTC" firstStartedPulling="2026-05-25 12:41:18.667492402 +0000 UTC m=+1714.325823785" lastFinishedPulling="2026-05-25 12:41:25.59614805 +0000 UTC m=+1721.254479436" observedRunningTime="2026-05-25 12:41:26.709815239 +0000 UTC m=+1722.368146643" watchObservedRunningTime="2026-05-25 12:41:26.711319426 +0000 UTC m=+1722.369650830"
May 25 12:41:27.697336 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:27.697302    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:27.697715 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:27.697344    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:27.698653 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:27.698624    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:41:28.699649 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:28.699607    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:41:33.705086 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:33.705057    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:41:33.705618 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:33.705593    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:41:43.706028 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:43.705985    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:41:53.706108 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:41:53.706068    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:42:03.706247 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:03.706204    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:42:13.706314 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:13.706278    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:42:23.705877 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:23.705837    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:42:33.705541 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:33.705498    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:42:39.951324 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:39.951229    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 25 12:42:49.952448 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:49.952418    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:42:54.356168 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:54.356123    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"]
May 25 12:42:54.356648 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:54.356482    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container" containerID="cri-o://05ccb5b944e3f89e2d6afbc84c720df81046cbd3a91906ee5ce5461372648f7f" gracePeriod=30
May 25 12:42:54.356648 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:54.356576    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kube-rbac-proxy" containerID="cri-o://7449349c406c7d11584835185366c3cdeac02915df38043536220bf46b7c4d13" gracePeriod=30
May 25 12:42:54.928383 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:54.928350    2562 generic.go:358] "Generic (PLEG): container finished" podID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerID="7449349c406c7d11584835185366c3cdeac02915df38043536220bf46b7c4d13" exitCode=2
May 25 12:42:54.928545 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:54.928426    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" event={"ID":"94fc6618-bf69-4a33-8784-9a084a8f8dee","Type":"ContainerDied","Data":"7449349c406c7d11584835185366c3cdeac02915df38043536220bf46b7c4d13"}
May 25 12:42:57.937822 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:57.937785    2562 generic.go:358] "Generic (PLEG): container finished" podID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerID="05ccb5b944e3f89e2d6afbc84c720df81046cbd3a91906ee5ce5461372648f7f" exitCode=0
May 25 12:42:57.938219 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:57.937832    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" event={"ID":"94fc6618-bf69-4a33-8784-9a084a8f8dee","Type":"ContainerDied","Data":"05ccb5b944e3f89e2d6afbc84c720df81046cbd3a91906ee5ce5461372648f7f"}
May 25 12:42:57.994464 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:57.994432    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:42:58.114643 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.114555    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h25x4\" (UniqueName: \"kubernetes.io/projected/94fc6618-bf69-4a33-8784-9a084a8f8dee-kube-api-access-h25x4\") pod \"94fc6618-bf69-4a33-8784-9a084a8f8dee\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") "
May 25 12:42:58.114643 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.114603    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94fc6618-bf69-4a33-8784-9a084a8f8dee-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"94fc6618-bf69-4a33-8784-9a084a8f8dee\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") "
May 25 12:42:58.114643 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.114633    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94fc6618-bf69-4a33-8784-9a084a8f8dee-proxy-tls\") pod \"94fc6618-bf69-4a33-8784-9a084a8f8dee\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") "
May 25 12:42:58.114855 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.114683    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94fc6618-bf69-4a33-8784-9a084a8f8dee-kserve-provision-location\") pod \"94fc6618-bf69-4a33-8784-9a084a8f8dee\" (UID: \"94fc6618-bf69-4a33-8784-9a084a8f8dee\") "
May 25 12:42:58.115092 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.115058    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fc6618-bf69-4a33-8784-9a084a8f8dee-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "94fc6618-bf69-4a33-8784-9a084a8f8dee" (UID: "94fc6618-bf69-4a33-8784-9a084a8f8dee"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:42:58.115220 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.115061    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fc6618-bf69-4a33-8784-9a084a8f8dee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94fc6618-bf69-4a33-8784-9a084a8f8dee" (UID: "94fc6618-bf69-4a33-8784-9a084a8f8dee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:42:58.116712 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.116687    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fc6618-bf69-4a33-8784-9a084a8f8dee-kube-api-access-h25x4" (OuterVolumeSpecName: "kube-api-access-h25x4") pod "94fc6618-bf69-4a33-8784-9a084a8f8dee" (UID: "94fc6618-bf69-4a33-8784-9a084a8f8dee"). InnerVolumeSpecName "kube-api-access-h25x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:42:58.116778 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.116705    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fc6618-bf69-4a33-8784-9a084a8f8dee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94fc6618-bf69-4a33-8784-9a084a8f8dee" (UID: "94fc6618-bf69-4a33-8784-9a084a8f8dee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:42:58.216161 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.216121    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94fc6618-bf69-4a33-8784-9a084a8f8dee-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:42:58.216161 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.216153    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h25x4\" (UniqueName: \"kubernetes.io/projected/94fc6618-bf69-4a33-8784-9a084a8f8dee-kube-api-access-h25x4\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:42:58.216161 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.216163    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94fc6618-bf69-4a33-8784-9a084a8f8dee-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:42:58.216418 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.216176    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94fc6618-bf69-4a33-8784-9a084a8f8dee-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:42:58.942642 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.942604    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g" event={"ID":"94fc6618-bf69-4a33-8784-9a084a8f8dee","Type":"ContainerDied","Data":"c4c2617363ce90d1d4a939ad284497f39bd64bf4511964c7801cfa642faf403c"}
May 25 12:42:58.943055 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.942655    2562 scope.go:117] "RemoveContainer" containerID="7449349c406c7d11584835185366c3cdeac02915df38043536220bf46b7c4d13"
May 25 12:42:58.943055 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.942665    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"
May 25 12:42:58.951139 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.951049    2562 scope.go:117] "RemoveContainer" containerID="05ccb5b944e3f89e2d6afbc84c720df81046cbd3a91906ee5ce5461372648f7f"
May 25 12:42:58.958756 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.958735    2562 scope.go:117] "RemoveContainer" containerID="819cc911107f018f8a0c571bffbd8cb8d8d92c9567baa5a4edbd05174c16d8e6"
May 25 12:42:58.963638 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.963612    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"]
May 25 12:42:58.966893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:42:58.966872    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-h7l8g"]
May 25 12:43:00.953625 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:43:00.953590    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" path="/var/lib/kubelet/pods/94fc6618-bf69-4a33-8784-9a084a8f8dee/volumes"
May 25 12:44:35.643735 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.643700    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"]
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.643936    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.643946    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.643965    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.643971    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.643979    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kube-rbac-proxy"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.643986    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kube-rbac-proxy"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.643994    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="storage-initializer"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.644000    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="storage-initializer"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.644006    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="storage-initializer"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.644012    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="storage-initializer"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.644018    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kube-rbac-proxy"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.644022    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kube-rbac-proxy"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.644060    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kube-rbac-proxy"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.644067    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kserve-container"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.644072    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="94fc6618-bf69-4a33-8784-9a084a8f8dee" containerName="kube-rbac-proxy"
May 25 12:44:35.644150 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.644080    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0ea7dc2-a634-4d88-a42d-26cce3db800b" containerName="kserve-container"
May 25 12:44:35.646152 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.646132    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.648763 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.648738    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:44:35.648877 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.648762    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
May 25 12:44:35.648877 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.648768    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
May 25 12:44:35.648877 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.648742    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 12:44:35.649951 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.649935    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:44:35.654766 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.654743    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"]
May 25 12:44:35.801913 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.801868    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01714674-077f-48b8-aa7d-eceda7c26416-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.801913 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.801914    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01714674-077f-48b8-aa7d-eceda7c26416-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.802129 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.801940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01714674-077f-48b8-aa7d-eceda7c26416-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.802129 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.802043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx65h\" (UniqueName: \"kubernetes.io/projected/01714674-077f-48b8-aa7d-eceda7c26416-kube-api-access-sx65h\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.902954 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.902861    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sx65h\" (UniqueName: \"kubernetes.io/projected/01714674-077f-48b8-aa7d-eceda7c26416-kube-api-access-sx65h\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.902954 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.902937    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01714674-077f-48b8-aa7d-eceda7c26416-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.902954 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.902956    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01714674-077f-48b8-aa7d-eceda7c26416-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.903219 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.902976    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01714674-077f-48b8-aa7d-eceda7c26416-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.903219 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:44:35.903086    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
May 25 12:44:35.903219 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:44:35.903163    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01714674-077f-48b8-aa7d-eceda7c26416-proxy-tls podName:01714674-077f-48b8-aa7d-eceda7c26416 nodeName:}" failed. No retries permitted until 2026-05-25 12:44:36.403141224 +0000 UTC m=+1912.061472610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/01714674-077f-48b8-aa7d-eceda7c26416-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" (UID: "01714674-077f-48b8-aa7d-eceda7c26416") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
May 25 12:44:35.903479 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.903445    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01714674-077f-48b8-aa7d-eceda7c26416-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.903703 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.903683    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01714674-077f-48b8-aa7d-eceda7c26416-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:35.913327 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:35.913304    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx65h\" (UniqueName: \"kubernetes.io/projected/01714674-077f-48b8-aa7d-eceda7c26416-kube-api-access-sx65h\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:36.405872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:36.405830    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01714674-077f-48b8-aa7d-eceda7c26416-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:36.408218 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:36.408193    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01714674-077f-48b8-aa7d-eceda7c26416-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:36.556600 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:36.556554    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:36.681510 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:36.681425    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"]
May 25 12:44:36.684327 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:44:36.684293    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01714674_077f_48b8_aa7d_eceda7c26416.slice/crio-274d2eb656ddd6e0296bc3cae733be961b2857dc2d8dd36512c388b63098a044 WatchSource:0}: Error finding container 274d2eb656ddd6e0296bc3cae733be961b2857dc2d8dd36512c388b63098a044: Status 404 returned error can't find the container with id 274d2eb656ddd6e0296bc3cae733be961b2857dc2d8dd36512c388b63098a044
May 25 12:44:36.686464 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:36.686449    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:44:37.192737 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:37.192698    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" event={"ID":"01714674-077f-48b8-aa7d-eceda7c26416","Type":"ContainerStarted","Data":"59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d"}
May 25 12:44:37.192737 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:37.192739    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" event={"ID":"01714674-077f-48b8-aa7d-eceda7c26416","Type":"ContainerStarted","Data":"274d2eb656ddd6e0296bc3cae733be961b2857dc2d8dd36512c388b63098a044"}
May 25 12:44:41.203314 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:41.203278    2562 generic.go:358] "Generic (PLEG): container finished" podID="01714674-077f-48b8-aa7d-eceda7c26416" containerID="59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d" exitCode=0
May 25 12:44:41.203751 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:41.203330    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" event={"ID":"01714674-077f-48b8-aa7d-eceda7c26416","Type":"ContainerDied","Data":"59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d"}
May 25 12:44:42.208044 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:42.208007    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" event={"ID":"01714674-077f-48b8-aa7d-eceda7c26416","Type":"ContainerStarted","Data":"03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7"}
May 25 12:44:42.208044 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:42.208051    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" event={"ID":"01714674-077f-48b8-aa7d-eceda7c26416","Type":"ContainerStarted","Data":"bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b"}
May 25 12:44:42.208576 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:42.208290    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:42.225525 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:42.225482    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podStartSLOduration=7.22546891 podStartE2EDuration="7.22546891s" podCreationTimestamp="2026-05-25 12:44:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:44:42.225232174 +0000 UTC m=+1917.883563595" watchObservedRunningTime="2026-05-25 12:44:42.22546891 +0000 UTC m=+1917.883800309"
May 25 12:44:43.210580 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:43.210551    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:43.211739 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:43.211713    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:44:44.212906 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:44.212868    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:44:49.217067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:49.217031    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:44:49.217685 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:49.217652    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:44:59.217608 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:44:59.217563    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:45:09.218311 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:45:09.218253    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:45:19.217801 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:45:19.217753    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:45:29.218427 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:45:29.218384    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:45:39.217898 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:45:39.217851    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:45:49.218020 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:45:49.217971    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:45:59.217897 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:45:59.217855    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:46:09.218905 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:09.218875    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:46:16.774777 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:16.774746    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"]
May 25 12:46:16.775191 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:16.775013    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" containerID="cri-o://bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b" gracePeriod=30
May 25 12:46:16.775191 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:16.775055    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kube-rbac-proxy" containerID="cri-o://03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7" gracePeriod=30
May 25 12:46:17.457854 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:17.457819    2562 generic.go:358] "Generic (PLEG): container finished" podID="01714674-077f-48b8-aa7d-eceda7c26416" containerID="03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7" exitCode=2
May 25 12:46:17.458027 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:17.457879    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" event={"ID":"01714674-077f-48b8-aa7d-eceda7c26416","Type":"ContainerDied","Data":"03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7"}
May 25 12:46:19.213455 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:19.213413    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
May 25 12:46:19.218402 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:19.218372    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 25 12:46:20.308827 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.308804    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:46:20.467238 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.467147    2562 generic.go:358] "Generic (PLEG): container finished" podID="01714674-077f-48b8-aa7d-eceda7c26416" containerID="bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b" exitCode=0
May 25 12:46:20.467238 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.467206    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" event={"ID":"01714674-077f-48b8-aa7d-eceda7c26416","Type":"ContainerDied","Data":"bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b"}
May 25 12:46:20.467238 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.467232    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"
May 25 12:46:20.467505 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.467246    2562 scope.go:117] "RemoveContainer" containerID="03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7"
May 25 12:46:20.467505 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.467233    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf" event={"ID":"01714674-077f-48b8-aa7d-eceda7c26416","Type":"ContainerDied","Data":"274d2eb656ddd6e0296bc3cae733be961b2857dc2d8dd36512c388b63098a044"}
May 25 12:46:20.474754 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.474734    2562 scope.go:117] "RemoveContainer" containerID="bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b"
May 25 12:46:20.481337 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.481320    2562 scope.go:117] "RemoveContainer" containerID="59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d"
May 25 12:46:20.487751 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.487733    2562 scope.go:117] "RemoveContainer" containerID="03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7"
May 25 12:46:20.488004 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:46:20.487967    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7\": container with ID starting with 03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7 not found: ID does not exist" containerID="03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7"
May 25 12:46:20.488044 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.488014    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7"} err="failed to get container status \"03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7\": rpc error: code = NotFound desc = could not find container \"03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7\": container with ID starting with 03c5eccb1e2a705d1c2c8eef5db9d5e1d998fbae2fc8a59ca335bf9e3f74cdd7 not found: ID does not exist"
May 25 12:46:20.488044 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.488034    2562 scope.go:117] "RemoveContainer" containerID="bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b"
May 25 12:46:20.488296 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:46:20.488277    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b\": container with ID starting with bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b not found: ID does not exist" containerID="bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b"
May 25 12:46:20.488371 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.488305    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b"} err="failed to get container status \"bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b\": rpc error: code = NotFound desc = could not find container \"bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b\": container with ID starting with bc606b27fa6d47d95f5fcb5be8c8d2cf0c41e4b1ad4fb28668a911fd2255555b not found: ID does not exist"
May 25 12:46:20.488371 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.488329    2562 scope.go:117] "RemoveContainer" containerID="59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d"
May 25 12:46:20.488566 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:46:20.488549    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d\": container with ID starting with 59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d not found: ID does not exist" containerID="59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d"
May 25 12:46:20.488604 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.488572    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d"} err="failed to get container status \"59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d\": rpc error: code = NotFound desc = could not find container \"59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d\": container with ID starting with 59887b3d337974eb611797ed06ccf6d813f5ac22dddd26dbcf44fa202dfa053d not found: ID does not exist"
May 25 12:46:20.489754 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.489739    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx65h\" (UniqueName: \"kubernetes.io/projected/01714674-077f-48b8-aa7d-eceda7c26416-kube-api-access-sx65h\") pod \"01714674-077f-48b8-aa7d-eceda7c26416\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") "
May 25 12:46:20.489810 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.489799    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01714674-077f-48b8-aa7d-eceda7c26416-kserve-provision-location\") pod \"01714674-077f-48b8-aa7d-eceda7c26416\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") "
May 25 12:46:20.489875 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.489831    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01714674-077f-48b8-aa7d-eceda7c26416-proxy-tls\") pod \"01714674-077f-48b8-aa7d-eceda7c26416\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") "
May 25 12:46:20.489875 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.489860    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01714674-077f-48b8-aa7d-eceda7c26416-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"01714674-077f-48b8-aa7d-eceda7c26416\" (UID: \"01714674-077f-48b8-aa7d-eceda7c26416\") "
May 25 12:46:20.490069 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.490045    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01714674-077f-48b8-aa7d-eceda7c26416-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01714674-077f-48b8-aa7d-eceda7c26416" (UID: "01714674-077f-48b8-aa7d-eceda7c26416"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:46:20.490255 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.490227    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01714674-077f-48b8-aa7d-eceda7c26416-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "01714674-077f-48b8-aa7d-eceda7c26416" (UID: "01714674-077f-48b8-aa7d-eceda7c26416"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:46:20.491719 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.491681    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01714674-077f-48b8-aa7d-eceda7c26416-kube-api-access-sx65h" (OuterVolumeSpecName: "kube-api-access-sx65h") pod "01714674-077f-48b8-aa7d-eceda7c26416" (UID: "01714674-077f-48b8-aa7d-eceda7c26416"). InnerVolumeSpecName "kube-api-access-sx65h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:46:20.491798 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.491735    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01714674-077f-48b8-aa7d-eceda7c26416-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01714674-077f-48b8-aa7d-eceda7c26416" (UID: "01714674-077f-48b8-aa7d-eceda7c26416"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:46:20.590317 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.590286    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01714674-077f-48b8-aa7d-eceda7c26416-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:46:20.590317 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.590311    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01714674-077f-48b8-aa7d-eceda7c26416-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:46:20.590502 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.590326    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01714674-077f-48b8-aa7d-eceda7c26416-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:46:20.590502 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.590337    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx65h\" (UniqueName: \"kubernetes.io/projected/01714674-077f-48b8-aa7d-eceda7c26416-kube-api-access-sx65h\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:46:20.789788 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.789754    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"]
May 25 12:46:20.792670 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.792634    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-8gtxf"]
May 25 12:46:20.954041 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:46:20.954006    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01714674-077f-48b8-aa7d-eceda7c26416" path="/var/lib/kubelet/pods/01714674-077f-48b8-aa7d-eceda7c26416/volumes"
May 25 12:48:03.344492 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.344457    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"]
May 25 12:48:03.344899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.344705    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kube-rbac-proxy"
May 25 12:48:03.344899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.344716    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kube-rbac-proxy"
May 25 12:48:03.344899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.344733    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="storage-initializer"
May 25 12:48:03.344899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.344739    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="storage-initializer"
May 25 12:48:03.344899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.344771    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container"
May 25 12:48:03.344899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.344778    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container"
May 25 12:48:03.344899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.344816    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kube-rbac-proxy"
May 25 12:48:03.344899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.344826    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="01714674-077f-48b8-aa7d-eceda7c26416" containerName="kserve-container"
May 25 12:48:03.347540 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.347525    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.350235 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.350208    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
May 25 12:48:03.350377 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.350232    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
May 25 12:48:03.350377 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.350250    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 12:48:03.350377 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.350309    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:48:03.350377 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.350327    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:48:03.355247 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.355222    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"]
May 25 12:48:03.506023 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.505980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e37273b-aef8-43c1-8730-4686c0f0fac0-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.506222 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.506054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwc98\" (UniqueName: \"kubernetes.io/projected/8e37273b-aef8-43c1-8730-4686c0f0fac0-kube-api-access-wwc98\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.506222 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.506097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e37273b-aef8-43c1-8730-4686c0f0fac0-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.506222 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.506130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e37273b-aef8-43c1-8730-4686c0f0fac0-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.606633 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.606544    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e37273b-aef8-43c1-8730-4686c0f0fac0-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.606633 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.606588    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e37273b-aef8-43c1-8730-4686c0f0fac0-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.606633 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.606626    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e37273b-aef8-43c1-8730-4686c0f0fac0-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.606855 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.606663    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wwc98\" (UniqueName: \"kubernetes.io/projected/8e37273b-aef8-43c1-8730-4686c0f0fac0-kube-api-access-wwc98\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.606979 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.606958    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e37273b-aef8-43c1-8730-4686c0f0fac0-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.607308 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.607287    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e37273b-aef8-43c1-8730-4686c0f0fac0-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.609045 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.609027    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e37273b-aef8-43c1-8730-4686c0f0fac0-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.614613 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.614582    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwc98\" (UniqueName: \"kubernetes.io/projected/8e37273b-aef8-43c1-8730-4686c0f0fac0-kube-api-access-wwc98\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.658552 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.658512    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:03.781585 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:03.781543    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"]
May 25 12:48:03.784428 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:48:03.784398    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e37273b_aef8_43c1_8730_4686c0f0fac0.slice/crio-334c10221d4c6466b6323dc1764a1e9bbb7662d3d948fa377ef728d50d4b9b2d WatchSource:0}: Error finding container 334c10221d4c6466b6323dc1764a1e9bbb7662d3d948fa377ef728d50d4b9b2d: Status 404 returned error can't find the container with id 334c10221d4c6466b6323dc1764a1e9bbb7662d3d948fa377ef728d50d4b9b2d
May 25 12:48:04.745116 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:04.745081    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" event={"ID":"8e37273b-aef8-43c1-8730-4686c0f0fac0","Type":"ContainerStarted","Data":"fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c"}
May 25 12:48:04.745116 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:04.745119    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" event={"ID":"8e37273b-aef8-43c1-8730-4686c0f0fac0","Type":"ContainerStarted","Data":"334c10221d4c6466b6323dc1764a1e9bbb7662d3d948fa377ef728d50d4b9b2d"}
May 25 12:48:07.753653 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:07.753616    2562 generic.go:358] "Generic (PLEG): container finished" podID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerID="fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c" exitCode=0
May 25 12:48:07.754067 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:07.753679    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" event={"ID":"8e37273b-aef8-43c1-8730-4686c0f0fac0","Type":"ContainerDied","Data":"fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c"}
May 25 12:48:28.824520 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:28.824488    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" event={"ID":"8e37273b-aef8-43c1-8730-4686c0f0fac0","Type":"ContainerStarted","Data":"ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06"}
May 25 12:48:29.828737 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:29.828699    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" event={"ID":"8e37273b-aef8-43c1-8730-4686c0f0fac0","Type":"ContainerStarted","Data":"099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523"}
May 25 12:48:29.829159 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:29.828958    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:29.848209 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:29.848158    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podStartSLOduration=5.8891098710000005 podStartE2EDuration="26.848144469s" podCreationTimestamp="2026-05-25 12:48:03 +0000 UTC" firstStartedPulling="2026-05-25 12:48:07.754774641 +0000 UTC m=+2123.413106025" lastFinishedPulling="2026-05-25 12:48:28.713809236 +0000 UTC m=+2144.372140623" observedRunningTime="2026-05-25 12:48:29.845909353 +0000 UTC m=+2145.504240759" watchObservedRunningTime="2026-05-25 12:48:29.848144469 +0000 UTC m=+2145.506475874"
May 25 12:48:30.831253 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:30.831217    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:30.832431 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:30.832404    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:48:31.833513 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:31.833473    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:48:36.839553 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:36.839480    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:48:36.840021 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:36.839995    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:48:46.840280 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:46.840217    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:48:56.840456 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:48:56.840415    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:49:06.840564 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:49:06.840527    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:49:16.840663 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:49:16.840618    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:49:26.840513 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:49:26.840471    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:49:36.840874 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:49:36.840833    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:49:43.950916 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:49:43.950868    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:49:53.951458 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:49:53.951411    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:50:03.492840 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.492773    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"]
May 25 12:50:03.493251 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.493199    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" containerID="cri-o://ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06" gracePeriod=30
May 25 12:50:03.493327 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.493233    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kube-rbac-proxy" containerID="cri-o://099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523" gracePeriod=30
May 25 12:50:03.617320 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.617289    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"]
May 25 12:50:03.620539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.620515    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.623072 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.623053    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
May 25 12:50:03.623199 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.623146    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
May 25 12:50:03.629496 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.629471    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"]
May 25 12:50:03.722330 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.722293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e670584b-4d41-4c30-8494-d4d2cc99ff0e-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.722515 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.722345    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e670584b-4d41-4c30-8494-d4d2cc99ff0e-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.722515 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.722432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.722515 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.722478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtxj4\" (UniqueName: \"kubernetes.io/projected/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kube-api-access-mtxj4\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.823507 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.823394    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mtxj4\" (UniqueName: \"kubernetes.io/projected/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kube-api-access-mtxj4\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.823507 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.823456    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e670584b-4d41-4c30-8494-d4d2cc99ff0e-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.823507 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.823497    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e670584b-4d41-4c30-8494-d4d2cc99ff0e-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.823811 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.823531    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.823811 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:50:03.823650    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
May 25 12:50:03.823811 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:50:03.823722    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e670584b-4d41-4c30-8494-d4d2cc99ff0e-proxy-tls podName:e670584b-4d41-4c30-8494-d4d2cc99ff0e nodeName:}" failed. No retries permitted until 2026-05-25 12:50:04.323701072 +0000 UTC m=+2239.982032475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e670584b-4d41-4c30-8494-d4d2cc99ff0e-proxy-tls") pod "isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" (UID: "e670584b-4d41-4c30-8494-d4d2cc99ff0e") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
May 25 12:50:03.823929 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.823877    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.824145 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.824125    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e670584b-4d41-4c30-8494-d4d2cc99ff0e-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.834684 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.834655    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtxj4\" (UniqueName: \"kubernetes.io/projected/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kube-api-access-mtxj4\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:03.950996 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:03.950950    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 12:50:04.092091 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:04.091975    2562 generic.go:358] "Generic (PLEG): container finished" podID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerID="099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523" exitCode=2
May 25 12:50:04.092091 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:04.092038    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" event={"ID":"8e37273b-aef8-43c1-8730-4686c0f0fac0","Type":"ContainerDied","Data":"099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523"}
May 25 12:50:04.327585 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:04.327551    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e670584b-4d41-4c30-8494-d4d2cc99ff0e-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:04.329873 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:04.329854    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e670584b-4d41-4c30-8494-d4d2cc99ff0e-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:04.530919 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:04.530880    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:04.660280 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:04.660130    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"]
May 25 12:50:04.662316 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:50:04.662285    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode670584b_4d41_4c30_8494_d4d2cc99ff0e.slice/crio-49b32c33f43b3b1e93f45f2103c08b48048a538e519f72229ffb7665b79f49f6 WatchSource:0}: Error finding container 49b32c33f43b3b1e93f45f2103c08b48048a538e519f72229ffb7665b79f49f6: Status 404 returned error can't find the container with id 49b32c33f43b3b1e93f45f2103c08b48048a538e519f72229ffb7665b79f49f6
May 25 12:50:04.664123 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:04.664107    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:50:05.096790 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:05.096756    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" event={"ID":"e670584b-4d41-4c30-8494-d4d2cc99ff0e","Type":"ContainerStarted","Data":"cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566"}
May 25 12:50:05.096944 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:05.096796    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" event={"ID":"e670584b-4d41-4c30-8494-d4d2cc99ff0e","Type":"ContainerStarted","Data":"49b32c33f43b3b1e93f45f2103c08b48048a538e519f72229ffb7665b79f49f6"}
May 25 12:50:06.834758 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:06.834694    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 25 12:50:08.597646 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.597580    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:50:08.661860 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.661826    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e37273b-aef8-43c1-8730-4686c0f0fac0-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"8e37273b-aef8-43c1-8730-4686c0f0fac0\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") "
May 25 12:50:08.662047 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.661884    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e37273b-aef8-43c1-8730-4686c0f0fac0-proxy-tls\") pod \"8e37273b-aef8-43c1-8730-4686c0f0fac0\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") "
May 25 12:50:08.662047 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.661901    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwc98\" (UniqueName: \"kubernetes.io/projected/8e37273b-aef8-43c1-8730-4686c0f0fac0-kube-api-access-wwc98\") pod \"8e37273b-aef8-43c1-8730-4686c0f0fac0\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") "
May 25 12:50:08.662344 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.662314    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e37273b-aef8-43c1-8730-4686c0f0fac0-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "8e37273b-aef8-43c1-8730-4686c0f0fac0" (UID: "8e37273b-aef8-43c1-8730-4686c0f0fac0"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:50:08.663958 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.663934    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e37273b-aef8-43c1-8730-4686c0f0fac0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e37273b-aef8-43c1-8730-4686c0f0fac0" (UID: "8e37273b-aef8-43c1-8730-4686c0f0fac0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:50:08.664047 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.663968    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e37273b-aef8-43c1-8730-4686c0f0fac0-kube-api-access-wwc98" (OuterVolumeSpecName: "kube-api-access-wwc98") pod "8e37273b-aef8-43c1-8730-4686c0f0fac0" (UID: "8e37273b-aef8-43c1-8730-4686c0f0fac0"). InnerVolumeSpecName "kube-api-access-wwc98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:50:08.762307 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.762229    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e37273b-aef8-43c1-8730-4686c0f0fac0-kserve-provision-location\") pod \"8e37273b-aef8-43c1-8730-4686c0f0fac0\" (UID: \"8e37273b-aef8-43c1-8730-4686c0f0fac0\") "
May 25 12:50:08.762485 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.762389    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e37273b-aef8-43c1-8730-4686c0f0fac0-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:50:08.762485 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.762405    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwc98\" (UniqueName: \"kubernetes.io/projected/8e37273b-aef8-43c1-8730-4686c0f0fac0-kube-api-access-wwc98\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:50:08.762485 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.762416    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e37273b-aef8-43c1-8730-4686c0f0fac0-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:50:08.762580 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.762536    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e37273b-aef8-43c1-8730-4686c0f0fac0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e37273b-aef8-43c1-8730-4686c0f0fac0" (UID: "8e37273b-aef8-43c1-8730-4686c0f0fac0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:50:08.863199 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:08.863112    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e37273b-aef8-43c1-8730-4686c0f0fac0-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:50:09.109033 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.108994    2562 generic.go:358] "Generic (PLEG): container finished" podID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerID="ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06" exitCode=0
May 25 12:50:09.109214 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.109081    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"
May 25 12:50:09.109214 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.109079    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" event={"ID":"8e37273b-aef8-43c1-8730-4686c0f0fac0","Type":"ContainerDied","Data":"ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06"}
May 25 12:50:09.109214 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.109117    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp" event={"ID":"8e37273b-aef8-43c1-8730-4686c0f0fac0","Type":"ContainerDied","Data":"334c10221d4c6466b6323dc1764a1e9bbb7662d3d948fa377ef728d50d4b9b2d"}
May 25 12:50:09.109214 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.109133    2562 scope.go:117] "RemoveContainer" containerID="099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523"
May 25 12:50:09.113489 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.111896    2562 generic.go:358] "Generic (PLEG): container finished" podID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerID="cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566" exitCode=0
May 25 12:50:09.113489 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.111967    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" event={"ID":"e670584b-4d41-4c30-8494-d4d2cc99ff0e","Type":"ContainerDied","Data":"cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566"}
May 25 12:50:09.117782 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.117755    2562 scope.go:117] "RemoveContainer" containerID="ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06"
May 25 12:50:09.125643 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.125556    2562 scope.go:117] "RemoveContainer" containerID="fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c"
May 25 12:50:09.127652 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.127630    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"]
May 25 12:50:09.130827 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.130809    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-8jfjp"]
May 25 12:50:09.134023 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.134000    2562 scope.go:117] "RemoveContainer" containerID="099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523"
May 25 12:50:09.134337 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:50:09.134317    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523\": container with ID starting with 099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523 not found: ID does not exist" containerID="099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523"
May 25 12:50:09.134436 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.134346    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523"} err="failed to get container status \"099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523\": rpc error: code = NotFound desc = could not find container \"099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523\": container with ID starting with 099e59cc7971dccb2ec8ce7574ef9f36049f399f0a1a5e06bf8c0f6dfcd16523 not found: ID does not exist"
May 25 12:50:09.134436 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.134367    2562 scope.go:117] "RemoveContainer" containerID="ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06"
May 25 12:50:09.134658 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:50:09.134637    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06\": container with ID starting with ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06 not found: ID does not exist" containerID="ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06"
May 25 12:50:09.134723 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.134667    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06"} err="failed to get container status \"ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06\": rpc error: code = NotFound desc = could not find container \"ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06\": container with ID starting with ab64a210c385c4c1e1f065dc8d919048ac7ffd5bb470094e7f4a44633a0bfd06 not found: ID does not exist"
May 25 12:50:09.134723 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.134685    2562 scope.go:117] "RemoveContainer" containerID="fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c"
May 25 12:50:09.134944 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:50:09.134926    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c\": container with ID starting with fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c not found: ID does not exist" containerID="fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c"
May 25 12:50:09.135004 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:09.134947    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c"} err="failed to get container status \"fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c\": rpc error: code = NotFound desc = could not find container \"fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c\": container with ID starting with fb88cff566694a5d765d0b4e7f752a101c8b7497e31db57621b351ee98ae4f7c not found: ID does not exist"
May 25 12:50:10.116504 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:10.116465    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" event={"ID":"e670584b-4d41-4c30-8494-d4d2cc99ff0e","Type":"ContainerStarted","Data":"8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b"}
May 25 12:50:10.116973 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:10.116520    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" event={"ID":"e670584b-4d41-4c30-8494-d4d2cc99ff0e","Type":"ContainerStarted","Data":"d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b"}
May 25 12:50:10.116973 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:10.116943    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:10.134411 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:10.134359    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podStartSLOduration=7.134346228 podStartE2EDuration="7.134346228s" podCreationTimestamp="2026-05-25 12:50:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:50:10.132691929 +0000 UTC m=+2245.791023333" watchObservedRunningTime="2026-05-25 12:50:10.134346228 +0000 UTC m=+2245.792677632"
May 25 12:50:10.953882 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:10.953851    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" path="/var/lib/kubelet/pods/8e37273b-aef8-43c1-8730-4686c0f0fac0/volumes"
May 25 12:50:11.120310 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:11.120279    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:11.121732 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:11.121703    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:50:12.123517 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:12.123474    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:50:17.127682 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:17.127655    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:50:17.128210 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:17.128185    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:50:27.128217 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:27.128173    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:50:37.128504 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:37.128468    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:50:47.128717 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:47.128680    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:50:57.128852 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:50:57.128813    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:51:07.128178 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:07.128139    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:51:17.128495 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:17.128457    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:51:22.950827 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:22.950790    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 12:51:32.954610 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:32.954579    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:51:33.771200 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.771152    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"]
May 25 12:51:33.771522 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.771468    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container" containerID="cri-o://d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b" gracePeriod=30
May 25 12:51:33.771658 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.771506    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kube-rbac-proxy" containerID="cri-o://8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b" gracePeriod=30
May 25 12:51:33.884079 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.884043    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"]
May 25 12:51:33.884375 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.884358    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kube-rbac-proxy"
May 25 12:51:33.884375 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.884374    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kube-rbac-proxy"
May 25 12:51:33.884475 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.884385    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="storage-initializer"
May 25 12:51:33.884475 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.884391    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="storage-initializer"
May 25 12:51:33.884475 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.884400    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container"
May 25 12:51:33.884475 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.884407    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container"
May 25 12:51:33.884475 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.884451    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kserve-container"
May 25 12:51:33.884475 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.884459    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e37273b-aef8-43c1-8730-4686c0f0fac0" containerName="kube-rbac-proxy"
May 25 12:51:33.887088 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.887065    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:33.889488 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.889468    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
May 25 12:51:33.889632 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.889615    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
May 25 12:51:33.897186 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.897160    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"]
May 25 12:51:33.950939 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.950906    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:33.951104 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.950953    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:33.951104 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.951021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fcrw\" (UniqueName: \"kubernetes.io/projected/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kube-api-access-7fcrw\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:33.951104 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:33.951084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.052287 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.052181    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.052287 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.052229    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7fcrw\" (UniqueName: \"kubernetes.io/projected/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kube-api-access-7fcrw\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.052742 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.052299    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.052742 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.052328    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.052742 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:51:34.052374    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
May 25 12:51:34.052742 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:51:34.052447    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-proxy-tls podName:9bcfe67a-fac3-4fb2-8a9f-1588c8946d31 nodeName:}" failed. No retries permitted until 2026-05-25 12:51:34.552426053 +0000 UTC m=+2330.210757448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-proxy-tls") pod "isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" (UID: "9bcfe67a-fac3-4fb2-8a9f-1588c8946d31") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
May 25 12:51:34.052881 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.052779    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.052930 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.052910    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.060402 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.060382    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fcrw\" (UniqueName: \"kubernetes.io/projected/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kube-api-access-7fcrw\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.354062 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.353954    2562 generic.go:358] "Generic (PLEG): container finished" podID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerID="8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b" exitCode=2
May 25 12:51:34.354062 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.354035    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" event={"ID":"e670584b-4d41-4c30-8494-d4d2cc99ff0e","Type":"ContainerDied","Data":"8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b"}
May 25 12:51:34.556033 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.555998    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.558480 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.558458    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.798184 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.798150    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:34.926558 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:34.926415    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"]
May 25 12:51:34.929214 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:51:34.929189    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcfe67a_fac3_4fb2_8a9f_1588c8946d31.slice/crio-652d4ab2762855d60897184ac632040d804043f8721057dfd5f2ca76660acd95 WatchSource:0}: Error finding container 652d4ab2762855d60897184ac632040d804043f8721057dfd5f2ca76660acd95: Status 404 returned error can't find the container with id 652d4ab2762855d60897184ac632040d804043f8721057dfd5f2ca76660acd95
May 25 12:51:35.358088 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:35.358047    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" event={"ID":"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31","Type":"ContainerStarted","Data":"6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708"}
May 25 12:51:35.358088 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:35.358089    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" event={"ID":"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31","Type":"ContainerStarted","Data":"652d4ab2762855d60897184ac632040d804043f8721057dfd5f2ca76660acd95"}
May 25 12:51:37.123878 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:37.123825    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
May 25 12:51:38.822800 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.822776    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:51:38.889430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.889330    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kserve-provision-location\") pod \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") "
May 25 12:51:38.889430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.889398    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e670584b-4d41-4c30-8494-d4d2cc99ff0e-proxy-tls\") pod \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") "
May 25 12:51:38.889430 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.889424    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtxj4\" (UniqueName: \"kubernetes.io/projected/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kube-api-access-mtxj4\") pod \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") "
May 25 12:51:38.889699 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.889497    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e670584b-4d41-4c30-8494-d4d2cc99ff0e-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\" (UID: \"e670584b-4d41-4c30-8494-d4d2cc99ff0e\") "
May 25 12:51:38.890409 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.890167    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e670584b-4d41-4c30-8494-d4d2cc99ff0e" (UID: "e670584b-4d41-4c30-8494-d4d2cc99ff0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:51:38.890409 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.890356    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e670584b-4d41-4c30-8494-d4d2cc99ff0e-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "e670584b-4d41-4c30-8494-d4d2cc99ff0e" (UID: "e670584b-4d41-4c30-8494-d4d2cc99ff0e"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:51:38.895681 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.895498    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e670584b-4d41-4c30-8494-d4d2cc99ff0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e670584b-4d41-4c30-8494-d4d2cc99ff0e" (UID: "e670584b-4d41-4c30-8494-d4d2cc99ff0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:51:38.896941 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.896916    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kube-api-access-mtxj4" (OuterVolumeSpecName: "kube-api-access-mtxj4") pod "e670584b-4d41-4c30-8494-d4d2cc99ff0e" (UID: "e670584b-4d41-4c30-8494-d4d2cc99ff0e"). InnerVolumeSpecName "kube-api-access-mtxj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:51:38.991138 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.991101    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e670584b-4d41-4c30-8494-d4d2cc99ff0e-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:51:38.991138 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.991135    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtxj4\" (UniqueName: \"kubernetes.io/projected/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kube-api-access-mtxj4\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:51:38.991346 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.991149    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e670584b-4d41-4c30-8494-d4d2cc99ff0e-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:51:38.991346 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:38.991164    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e670584b-4d41-4c30-8494-d4d2cc99ff0e-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:51:39.370568 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.370536    2562 generic.go:358] "Generic (PLEG): container finished" podID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerID="d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b" exitCode=0
May 25 12:51:39.370751 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.370645    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"
May 25 12:51:39.370751 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.370642    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" event={"ID":"e670584b-4d41-4c30-8494-d4d2cc99ff0e","Type":"ContainerDied","Data":"d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b"}
May 25 12:51:39.370751 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.370745    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d" event={"ID":"e670584b-4d41-4c30-8494-d4d2cc99ff0e","Type":"ContainerDied","Data":"49b32c33f43b3b1e93f45f2103c08b48048a538e519f72229ffb7665b79f49f6"}
May 25 12:51:39.370899 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.370769    2562 scope.go:117] "RemoveContainer" containerID="8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b"
May 25 12:51:39.372026 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.372000    2562 generic.go:358] "Generic (PLEG): container finished" podID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerID="6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708" exitCode=0
May 25 12:51:39.372121 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.372042    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" event={"ID":"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31","Type":"ContainerDied","Data":"6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708"}
May 25 12:51:39.378542 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.378453    2562 scope.go:117] "RemoveContainer" containerID="d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b"
May 25 12:51:39.385872 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.385848    2562 scope.go:117] "RemoveContainer" containerID="cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566"
May 25 12:51:39.393645 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.393624    2562 scope.go:117] "RemoveContainer" containerID="8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b"
May 25 12:51:39.394137 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:51:39.394094    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b\": container with ID starting with 8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b not found: ID does not exist" containerID="8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b"
May 25 12:51:39.394508 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.394202    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b"} err="failed to get container status \"8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b\": rpc error: code = NotFound desc = could not find container \"8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b\": container with ID starting with 8ab43f55e637e2dc00f9a2979ef6d6ac9dd00c1d416f3bd61594726c20fd7a7b not found: ID does not exist"
May 25 12:51:39.394508 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.394231    2562 scope.go:117] "RemoveContainer" containerID="d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b"
May 25 12:51:39.394881 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:51:39.394840    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b\": container with ID starting with d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b not found: ID does not exist" containerID="d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b"
May 25 12:51:39.394967 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.394878    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b"} err="failed to get container status \"d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b\": rpc error: code = NotFound desc = could not find container \"d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b\": container with ID starting with d7e7390f4a4c275f0ba0d72e58f9819d9fb6d6fae0ac07f4e355fb7e7e965c6b not found: ID does not exist"
May 25 12:51:39.394967 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.394915    2562 scope.go:117] "RemoveContainer" containerID="cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566"
May 25 12:51:39.395717 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:51:39.395692    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566\": container with ID starting with cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566 not found: ID does not exist" containerID="cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566"
May 25 12:51:39.395828 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.395723    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566"} err="failed to get container status \"cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566\": rpc error: code = NotFound desc = could not find container \"cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566\": container with ID starting with cc6c80eebafd95433202c6d95ba92c416b4d29e7b5135a473f0f6c290aa94566 not found: ID does not exist"
May 25 12:51:39.406055 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.406019    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"]
May 25 12:51:39.409968 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:39.409940    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-89f6d"]
May 25 12:51:40.376852 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:40.376808    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" event={"ID":"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31","Type":"ContainerStarted","Data":"2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b"}
May 25 12:51:40.377312 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:40.376862    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" event={"ID":"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31","Type":"ContainerStarted","Data":"1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252"}
May 25 12:51:40.377312 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:40.377154    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:40.377312 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:40.377181    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:40.378456 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:40.378429    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:51:40.395716 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:40.395670    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podStartSLOduration=7.395655967 podStartE2EDuration="7.395655967s" podCreationTimestamp="2026-05-25 12:51:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:51:40.394341393 +0000 UTC m=+2336.052672798" watchObservedRunningTime="2026-05-25 12:51:40.395655967 +0000 UTC m=+2336.053987371"
May 25 12:51:40.954710 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:40.954676    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" path="/var/lib/kubelet/pods/e670584b-4d41-4c30-8494-d4d2cc99ff0e/volumes"
May 25 12:51:41.381040 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:41.380994    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:51:46.385024 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:46.384995    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:51:46.385594 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:46.385564    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:51:56.386065 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:51:56.386020    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:52:06.385889 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:52:06.385847    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:52:16.385589 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:52:16.385546    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:52:26.386293 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:52:26.386239    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:52:36.386227 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:52:36.386189    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:52:46.386349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:52:46.386307    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:52:54.955757 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:52:54.955724    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:53:03.993900 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:03.993865    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"]
May 25 12:53:03.994321 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:03.994161    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" containerID="cri-o://1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252" gracePeriod=30
May 25 12:53:03.994321 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:03.994214    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kube-rbac-proxy" containerID="cri-o://2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b" gracePeriod=30
May 25 12:53:04.111856 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.111824    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"]
May 25 12:53:04.112149 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.112132    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="storage-initializer"
May 25 12:53:04.112228 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.112153    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="storage-initializer"
May 25 12:53:04.112228 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.112168    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container"
May 25 12:53:04.112228 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.112177    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container"
May 25 12:53:04.112228 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.112198    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kube-rbac-proxy"
May 25 12:53:04.112228 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.112207    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kube-rbac-proxy"
May 25 12:53:04.112494 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.112293    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kube-rbac-proxy"
May 25 12:53:04.112494 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.112305    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="e670584b-4d41-4c30-8494-d4d2cc99ff0e" containerName="kserve-container"
May 25 12:53:04.115055 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.115032    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.117332 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.117309    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
May 25 12:53:04.117438 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.117348    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
May 25 12:53:04.124857 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.124831    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"]
May 25 12:53:04.211520 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.211477    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3261ba8-1b34-4522-81b0-32002bd13976-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.211702 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.211555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3261ba8-1b34-4522-81b0-32002bd13976-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.211702 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.211576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3261ba8-1b34-4522-81b0-32002bd13976-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.211702 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.211594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qh5w\" (UniqueName: \"kubernetes.io/projected/c3261ba8-1b34-4522-81b0-32002bd13976-kube-api-access-7qh5w\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.312174 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.312080    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3261ba8-1b34-4522-81b0-32002bd13976-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.312174 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.312145    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3261ba8-1b34-4522-81b0-32002bd13976-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.312465 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.312176    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3261ba8-1b34-4522-81b0-32002bd13976-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.312465 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.312206    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7qh5w\" (UniqueName: \"kubernetes.io/projected/c3261ba8-1b34-4522-81b0-32002bd13976-kube-api-access-7qh5w\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.312465 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:53:04.312332    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
May 25 12:53:04.312465 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:53:04.312436    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3261ba8-1b34-4522-81b0-32002bd13976-proxy-tls podName:c3261ba8-1b34-4522-81b0-32002bd13976 nodeName:}" failed. No retries permitted until 2026-05-25 12:53:04.812410536 +0000 UTC m=+2420.470741922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c3261ba8-1b34-4522-81b0-32002bd13976-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" (UID: "c3261ba8-1b34-4522-81b0-32002bd13976") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
May 25 12:53:04.312704 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.312590    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3261ba8-1b34-4522-81b0-32002bd13976-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.312864 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.312844    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3261ba8-1b34-4522-81b0-32002bd13976-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.320210 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.320189    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qh5w\" (UniqueName: \"kubernetes.io/projected/c3261ba8-1b34-4522-81b0-32002bd13976-kube-api-access-7qh5w\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.600145 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.600060    2562 generic.go:358] "Generic (PLEG): container finished" podID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerID="2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b" exitCode=2
May 25 12:53:04.600145 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.600136    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" event={"ID":"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31","Type":"ContainerDied","Data":"2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b"}
May 25 12:53:04.817601 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.817555    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3261ba8-1b34-4522-81b0-32002bd13976-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.819883 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.819850    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3261ba8-1b34-4522-81b0-32002bd13976-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:04.953952 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:04.953865    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 12:53:05.024912 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:05.024866    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:05.144848 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:05.144817    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"]
May 25 12:53:05.147731 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:53:05.147685    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3261ba8_1b34_4522_81b0_32002bd13976.slice/crio-d02270ba2a11a99a5d5e9bc491bc50f17287315d59e93da10906ae54ffaa761a WatchSource:0}: Error finding container d02270ba2a11a99a5d5e9bc491bc50f17287315d59e93da10906ae54ffaa761a: Status 404 returned error can't find the container with id d02270ba2a11a99a5d5e9bc491bc50f17287315d59e93da10906ae54ffaa761a
May 25 12:53:05.604214 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:05.604175    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" event={"ID":"c3261ba8-1b34-4522-81b0-32002bd13976","Type":"ContainerStarted","Data":"cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e"}
May 25 12:53:05.604214 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:05.604216    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" event={"ID":"c3261ba8-1b34-4522-81b0-32002bd13976","Type":"ContainerStarted","Data":"d02270ba2a11a99a5d5e9bc491bc50f17287315d59e93da10906ae54ffaa761a"}
May 25 12:53:06.381802 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:06.381751    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 25 12:53:09.335179 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.335155    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:53:09.455432 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.455396    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fcrw\" (UniqueName: \"kubernetes.io/projected/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kube-api-access-7fcrw\") pod \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") "
May 25 12:53:09.455623 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.455463    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kserve-provision-location\") pod \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") "
May 25 12:53:09.455693 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.455630    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") "
May 25 12:53:09.455693 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.455672    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-proxy-tls\") pod \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\" (UID: \"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31\") "
May 25 12:53:09.455831 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.455809    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" (UID: "9bcfe67a-fac3-4fb2-8a9f-1588c8946d31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:53:09.456012 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.455987    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" (UID: "9bcfe67a-fac3-4fb2-8a9f-1588c8946d31"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:53:09.457574 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.457522    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kube-api-access-7fcrw" (OuterVolumeSpecName: "kube-api-access-7fcrw") pod "9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" (UID: "9bcfe67a-fac3-4fb2-8a9f-1588c8946d31"). InnerVolumeSpecName "kube-api-access-7fcrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:53:09.457690 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.457669    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" (UID: "9bcfe67a-fac3-4fb2-8a9f-1588c8946d31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:53:09.556968 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.556921    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fcrw\" (UniqueName: \"kubernetes.io/projected/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kube-api-access-7fcrw\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:53:09.556968 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.556965    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:53:09.556968 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.556982    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:53:09.557191 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.556999    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:53:09.615985 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.615953    2562 generic.go:358] "Generic (PLEG): container finished" podID="c3261ba8-1b34-4522-81b0-32002bd13976" containerID="cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e" exitCode=0
May 25 12:53:09.616155 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.616032    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" event={"ID":"c3261ba8-1b34-4522-81b0-32002bd13976","Type":"ContainerDied","Data":"cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e"}
May 25 12:53:09.617649 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.617624    2562 generic.go:358] "Generic (PLEG): container finished" podID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerID="1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252" exitCode=0
May 25 12:53:09.617737 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.617655    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" event={"ID":"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31","Type":"ContainerDied","Data":"1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252"}
May 25 12:53:09.617737 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.617686    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl" event={"ID":"9bcfe67a-fac3-4fb2-8a9f-1588c8946d31","Type":"ContainerDied","Data":"652d4ab2762855d60897184ac632040d804043f8721057dfd5f2ca76660acd95"}
May 25 12:53:09.617737 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.617705    2562 scope.go:117] "RemoveContainer" containerID="2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b"
May 25 12:53:09.617868 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.617705    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"
May 25 12:53:09.627483 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.627452    2562 scope.go:117] "RemoveContainer" containerID="1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252"
May 25 12:53:09.635887 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.635867    2562 scope.go:117] "RemoveContainer" containerID="6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708"
May 25 12:53:09.642700 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.642678    2562 scope.go:117] "RemoveContainer" containerID="2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b"
May 25 12:53:09.643087 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:53:09.643037    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b\": container with ID starting with 2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b not found: ID does not exist" containerID="2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b"
May 25 12:53:09.643189 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.643079    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b"} err="failed to get container status \"2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b\": rpc error: code = NotFound desc = could not find container \"2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b\": container with ID starting with 2f161f6a0a745074fb6289e0dc59b9d13420aed5c75b07905e7cfd4149beb65b not found: ID does not exist"
May 25 12:53:09.643189 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.643106    2562 scope.go:117] "RemoveContainer" containerID="1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252"
May 25 12:53:09.643586 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:53:09.643526    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252\": container with ID starting with 1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252 not found: ID does not exist" containerID="1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252"
May 25 12:53:09.643724 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.643557    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252"} err="failed to get container status \"1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252\": rpc error: code = NotFound desc = could not find container \"1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252\": container with ID starting with 1b1bf445362618b5639895cd8614c020f71f4a0201a8f8a2e1f872e99696f252 not found: ID does not exist"
May 25 12:53:09.643724 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.643718    2562 scope.go:117] "RemoveContainer" containerID="6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708"
May 25 12:53:09.644004 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:53:09.643986    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708\": container with ID starting with 6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708 not found: ID does not exist" containerID="6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708"
May 25 12:53:09.644079 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.644010    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708"} err="failed to get container status \"6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708\": rpc error: code = NotFound desc = could not find container \"6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708\": container with ID starting with 6f7346c616b2c4f2428057cf53b120839146718cff3b2120c1ff7b8d1be58708 not found: ID does not exist"
May 25 12:53:09.645102 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.645082    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"]
May 25 12:53:09.647645 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:09.647625    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-rm4gl"]
May 25 12:53:10.622447 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:10.622415    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" event={"ID":"c3261ba8-1b34-4522-81b0-32002bd13976","Type":"ContainerStarted","Data":"8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f"}
May 25 12:53:10.622890 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:10.622453    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" event={"ID":"c3261ba8-1b34-4522-81b0-32002bd13976","Type":"ContainerStarted","Data":"35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de"}
May 25 12:53:10.622890 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:10.622673    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:10.641941 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:10.641898    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podStartSLOduration=6.641885365 podStartE2EDuration="6.641885365s" podCreationTimestamp="2026-05-25 12:53:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:53:10.640852803 +0000 UTC m=+2426.299184243" watchObservedRunningTime="2026-05-25 12:53:10.641885365 +0000 UTC m=+2426.300216774"
May 25 12:53:10.958911 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:10.956009    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" path="/var/lib/kubelet/pods/9bcfe67a-fac3-4fb2-8a9f-1588c8946d31/volumes"
May 25 12:53:11.626648 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:11.626607    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:17.633866 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:17.633837    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:53:47.634792 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:47.634750    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 25 12:53:57.634917 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:53:57.634874    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 25 12:54:07.634578 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:07.634532    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 25 12:54:17.634704 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:17.634662    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 25 12:54:24.952235 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:24.952196    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 25 12:54:34.955825 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:34.955730    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:54:44.208305 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.208253    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"]
May 25 12:54:44.208734 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.208680    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container" containerID="cri-o://35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de" gracePeriod=30
May 25 12:54:44.208804 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.208735    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kube-rbac-proxy" containerID="cri-o://8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f" gracePeriod=30
May 25 12:54:44.323635 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.323603    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"]
May 25 12:54:44.323890 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.323877    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container"
May 25 12:54:44.323939 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.323895    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container"
May 25 12:54:44.323939 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.323918    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="storage-initializer"
May 25 12:54:44.323939 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.323924    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="storage-initializer"
May 25 12:54:44.323939 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.323931    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kube-rbac-proxy"
May 25 12:54:44.323939 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.323938    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kube-rbac-proxy"
May 25 12:54:44.324083 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.323984    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kserve-container"
May 25 12:54:44.324083 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.323994    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="9bcfe67a-fac3-4fb2-8a9f-1588c8946d31" containerName="kube-rbac-proxy"
May 25 12:54:44.326801 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.326784    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.329357 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.329339    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
May 25 12:54:44.329406 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.329363    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
May 25 12:54:44.340883 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.340855    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"]
May 25 12:54:44.448562 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.448520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0df651b-136c-415b-ba5f-07c28ec6d624-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.448729 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.448573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkkjb\" (UniqueName: \"kubernetes.io/projected/c0df651b-136c-415b-ba5f-07c28ec6d624-kube-api-access-mkkjb\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.448729 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.448604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0df651b-136c-415b-ba5f-07c28ec6d624-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.448729 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.448670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0df651b-136c-415b-ba5f-07c28ec6d624-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.549856 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.549824    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0df651b-136c-415b-ba5f-07c28ec6d624-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.550041 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.549904    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0df651b-136c-415b-ba5f-07c28ec6d624-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.550041 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.549939    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0df651b-136c-415b-ba5f-07c28ec6d624-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.550041 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.549968    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mkkjb\" (UniqueName: \"kubernetes.io/projected/c0df651b-136c-415b-ba5f-07c28ec6d624-kube-api-access-mkkjb\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.550372 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.550346    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0df651b-136c-415b-ba5f-07c28ec6d624-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.550502 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.550483    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0df651b-136c-415b-ba5f-07c28ec6d624-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.552469 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.552448    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0df651b-136c-415b-ba5f-07c28ec6d624-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.557961 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.557932    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkkjb\" (UniqueName: \"kubernetes.io/projected/c0df651b-136c-415b-ba5f-07c28ec6d624-kube-api-access-mkkjb\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.642956 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.642921    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:44.769499 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.769458    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"]
May 25 12:54:44.772424 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:54:44.772397    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0df651b_136c_415b_ba5f_07c28ec6d624.slice/crio-29661e4750e1417fcbd7c29d492699fbcde07dfe29c866206cacd02a0579bbcf WatchSource:0}: Error finding container 29661e4750e1417fcbd7c29d492699fbcde07dfe29c866206cacd02a0579bbcf: Status 404 returned error can't find the container with id 29661e4750e1417fcbd7c29d492699fbcde07dfe29c866206cacd02a0579bbcf
May 25 12:54:44.877153 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.877118    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" event={"ID":"c0df651b-136c-415b-ba5f-07c28ec6d624","Type":"ContainerStarted","Data":"eebcd86cb5ee46028d1d30d7ed991c9e021635e9c273d3dd3405f31e662248a4"}
May 25 12:54:44.877345 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.877161    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" event={"ID":"c0df651b-136c-415b-ba5f-07c28ec6d624","Type":"ContainerStarted","Data":"29661e4750e1417fcbd7c29d492699fbcde07dfe29c866206cacd02a0579bbcf"}
May 25 12:54:44.878967 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.878943    2562 generic.go:358] "Generic (PLEG): container finished" podID="c3261ba8-1b34-4522-81b0-32002bd13976" containerID="8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f" exitCode=2
May 25 12:54:44.879066 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.878976    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" event={"ID":"c3261ba8-1b34-4522-81b0-32002bd13976","Type":"ContainerDied","Data":"8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f"}
May 25 12:54:44.952744 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:44.952697    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 25 12:54:47.629888 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:47.629844    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
May 25 12:54:48.890452 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:48.890364    2562 generic.go:358] "Generic (PLEG): container finished" podID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerID="eebcd86cb5ee46028d1d30d7ed991c9e021635e9c273d3dd3405f31e662248a4" exitCode=0
May 25 12:54:48.890452 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:48.890440    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" event={"ID":"c0df651b-136c-415b-ba5f-07c28ec6d624","Type":"ContainerDied","Data":"eebcd86cb5ee46028d1d30d7ed991c9e021635e9c273d3dd3405f31e662248a4"}
May 25 12:54:49.352751 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.352727    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:54:49.490244 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.490142    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qh5w\" (UniqueName: \"kubernetes.io/projected/c3261ba8-1b34-4522-81b0-32002bd13976-kube-api-access-7qh5w\") pod \"c3261ba8-1b34-4522-81b0-32002bd13976\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") "
May 25 12:54:49.490244 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.490193    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3261ba8-1b34-4522-81b0-32002bd13976-proxy-tls\") pod \"c3261ba8-1b34-4522-81b0-32002bd13976\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") "
May 25 12:54:49.490244 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.490217    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3261ba8-1b34-4522-81b0-32002bd13976-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"c3261ba8-1b34-4522-81b0-32002bd13976\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") "
May 25 12:54:49.490485 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.490252    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3261ba8-1b34-4522-81b0-32002bd13976-kserve-provision-location\") pod \"c3261ba8-1b34-4522-81b0-32002bd13976\" (UID: \"c3261ba8-1b34-4522-81b0-32002bd13976\") "
May 25 12:54:49.490640 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.490612    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3261ba8-1b34-4522-81b0-32002bd13976-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "c3261ba8-1b34-4522-81b0-32002bd13976" (UID: "c3261ba8-1b34-4522-81b0-32002bd13976"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:54:49.490779 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.490644    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3261ba8-1b34-4522-81b0-32002bd13976-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3261ba8-1b34-4522-81b0-32002bd13976" (UID: "c3261ba8-1b34-4522-81b0-32002bd13976"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:54:49.492349 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.492326    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3261ba8-1b34-4522-81b0-32002bd13976-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3261ba8-1b34-4522-81b0-32002bd13976" (UID: "c3261ba8-1b34-4522-81b0-32002bd13976"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:54:49.492462 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.492436    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3261ba8-1b34-4522-81b0-32002bd13976-kube-api-access-7qh5w" (OuterVolumeSpecName: "kube-api-access-7qh5w") pod "c3261ba8-1b34-4522-81b0-32002bd13976" (UID: "c3261ba8-1b34-4522-81b0-32002bd13976"). InnerVolumeSpecName "kube-api-access-7qh5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:54:49.590943 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.590910    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qh5w\" (UniqueName: \"kubernetes.io/projected/c3261ba8-1b34-4522-81b0-32002bd13976-kube-api-access-7qh5w\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:54:49.590943 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.590940    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3261ba8-1b34-4522-81b0-32002bd13976-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:54:49.590943 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.590952    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3261ba8-1b34-4522-81b0-32002bd13976-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:54:49.591165 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.590962    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3261ba8-1b34-4522-81b0-32002bd13976-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:54:49.900514 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.900417    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" event={"ID":"c0df651b-136c-415b-ba5f-07c28ec6d624","Type":"ContainerStarted","Data":"9f92150a129efa6e2cb53d251371d140559fef16bebfbf1a85dffb8fe93fd7ed"}
May 25 12:54:49.900514 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.900462    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" event={"ID":"c0df651b-136c-415b-ba5f-07c28ec6d624","Type":"ContainerStarted","Data":"b8db2813f5db1381780a1987b1bdc046817ca75b7e56bd51d5dc9e62e46e7f81"}
May 25 12:54:49.901020 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.900828    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:49.901020 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.900869    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:54:49.902083 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.902058    2562 generic.go:358] "Generic (PLEG): container finished" podID="c3261ba8-1b34-4522-81b0-32002bd13976" containerID="35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de" exitCode=0
May 25 12:54:49.902187 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.902090    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" event={"ID":"c3261ba8-1b34-4522-81b0-32002bd13976","Type":"ContainerDied","Data":"35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de"}
May 25 12:54:49.902187 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.902108    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96" event={"ID":"c3261ba8-1b34-4522-81b0-32002bd13976","Type":"ContainerDied","Data":"d02270ba2a11a99a5d5e9bc491bc50f17287315d59e93da10906ae54ffaa761a"}
May 25 12:54:49.902187 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.902123    2562 scope.go:117] "RemoveContainer" containerID="8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f"
May 25 12:54:49.902187 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.902141    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"
May 25 12:54:49.910185 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.910169    2562 scope.go:117] "RemoveContainer" containerID="35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de"
May 25 12:54:49.918184 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.918164    2562 scope.go:117] "RemoveContainer" containerID="cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e"
May 25 12:54:49.920764 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.920715    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" podStartSLOduration=5.920698899 podStartE2EDuration="5.920698899s" podCreationTimestamp="2026-05-25 12:54:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:54:49.920397421 +0000 UTC m=+2525.578728828" watchObservedRunningTime="2026-05-25 12:54:49.920698899 +0000 UTC m=+2525.579030305"
May 25 12:54:49.926526 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.926509    2562 scope.go:117] "RemoveContainer" containerID="8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f"
May 25 12:54:49.926815 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:54:49.926796    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f\": container with ID starting with 8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f not found: ID does not exist" containerID="8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f"
May 25 12:54:49.926887 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.926828    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f"} err="failed to get container status \"8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f\": rpc error: code = NotFound desc = could not find container \"8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f\": container with ID starting with 8abe4547aa7429c2051d7830a63de362ac1bb5c914c86b1c7c95b9e43becc70f not found: ID does not exist"
May 25 12:54:49.926887 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.926854    2562 scope.go:117] "RemoveContainer" containerID="35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de"
May 25 12:54:49.927088 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:54:49.927070    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de\": container with ID starting with 35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de not found: ID does not exist" containerID="35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de"
May 25 12:54:49.927128 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.927094    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de"} err="failed to get container status \"35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de\": rpc error: code = NotFound desc = could not find container \"35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de\": container with ID starting with 35e05fd64870971023293442dc501fed41aa889ae71429c41e8fb6787eb8f6de not found: ID does not exist"
May 25 12:54:49.927128 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.927110    2562 scope.go:117] "RemoveContainer" containerID="cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e"
May 25 12:54:49.927334 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:54:49.927316    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e\": container with ID starting with cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e not found: ID does not exist" containerID="cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e"
May 25 12:54:49.927394 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.927338    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e"} err="failed to get container status \"cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e\": rpc error: code = NotFound desc = could not find container \"cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e\": container with ID starting with cb2deb498bae7709a8b871556608891f5f61750f12cb9a7fe643727c0399436e not found: ID does not exist"
May 25 12:54:49.935141 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.935118    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"]
May 25 12:54:49.940542 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:49.940520    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-82c96"]
May 25 12:54:50.956394 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:50.956360    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" path="/var/lib/kubelet/pods/c3261ba8-1b34-4522-81b0-32002bd13976/volumes"
May 25 12:54:55.910584 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:54:55.910552    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:55:25.912088 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:55:25.912045    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
May 25 12:55:35.911454 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:55:35.911415    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
May 25 12:55:45.911186 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:55:45.911141    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
May 25 12:55:55.911715 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:55:55.911675    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
May 25 12:56:05.914945 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:05.914865    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:56:14.425886 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.425849    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"]
May 25 12:56:14.426285 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.426184    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kserve-container" containerID="cri-o://b8db2813f5db1381780a1987b1bdc046817ca75b7e56bd51d5dc9e62e46e7f81" gracePeriod=30
May 25 12:56:14.426285 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.426227    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kube-rbac-proxy" containerID="cri-o://9f92150a129efa6e2cb53d251371d140559fef16bebfbf1a85dffb8fe93fd7ed" gracePeriod=30
May 25 12:56:14.533504 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.533469    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"]
May 25 12:56:14.533781 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.533769    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container"
May 25 12:56:14.533826 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.533783    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container"
May 25 12:56:14.533826 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.533797    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="storage-initializer"
May 25 12:56:14.533826 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.533805    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="storage-initializer"
May 25 12:56:14.533826 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.533819    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kube-rbac-proxy"
May 25 12:56:14.533826 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.533825    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kube-rbac-proxy"
May 25 12:56:14.533977 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.533876    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kserve-container"
May 25 12:56:14.533977 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.533889    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="c3261ba8-1b34-4522-81b0-32002bd13976" containerName="kube-rbac-proxy"
May 25 12:56:14.536979 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.536958    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.539378 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.539355    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
May 25 12:56:14.539536 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.539516    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
May 25 12:56:14.546277 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.546234    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"]
May 25 12:56:14.591518 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.591480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.591518 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.591521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.591733 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.591560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.591733 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.591582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7kgl\" (UniqueName: \"kubernetes.io/projected/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kube-api-access-w7kgl\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.692252 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.692152    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.692252 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.692200    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.692252 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.692227    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.692252 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.692245    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w7kgl\" (UniqueName: \"kubernetes.io/projected/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kube-api-access-w7kgl\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.692722 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.692695    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.692949 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.692932    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.694674 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.694658    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.700028 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.700005    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7kgl\" (UniqueName: \"kubernetes.io/projected/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kube-api-access-w7kgl\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.847995 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.847957    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:14.971228 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.971204    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"]
May 25 12:56:14.973279 ip-10-0-141-91 kubenswrapper[2562]: W0525 12:56:14.973240    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24a6a56_ac9a_4654_ae98_10ac72eb6bf7.slice/crio-25b9c67a9414273ae8bc2bc0e5b548500be082d7bd5c688a708f21bcce888c20 WatchSource:0}: Error finding container 25b9c67a9414273ae8bc2bc0e5b548500be082d7bd5c688a708f21bcce888c20: Status 404 returned error can't find the container with id 25b9c67a9414273ae8bc2bc0e5b548500be082d7bd5c688a708f21bcce888c20
May 25 12:56:14.974923 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:14.974907    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:56:15.126668 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:15.126635    2562 generic.go:358] "Generic (PLEG): container finished" podID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerID="9f92150a129efa6e2cb53d251371d140559fef16bebfbf1a85dffb8fe93fd7ed" exitCode=2
May 25 12:56:15.126837 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:15.126713    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" event={"ID":"c0df651b-136c-415b-ba5f-07c28ec6d624","Type":"ContainerDied","Data":"9f92150a129efa6e2cb53d251371d140559fef16bebfbf1a85dffb8fe93fd7ed"}
May 25 12:56:15.128098 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:15.128076    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" event={"ID":"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7","Type":"ContainerStarted","Data":"cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a"}
May 25 12:56:15.128211 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:15.128107    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" event={"ID":"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7","Type":"ContainerStarted","Data":"25b9c67a9414273ae8bc2bc0e5b548500be082d7bd5c688a708f21bcce888c20"}
May 25 12:56:15.905797 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:15.905754    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
May 25 12:56:15.911756 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:15.911728    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
May 25 12:56:19.139522 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.139487    2562 generic.go:358] "Generic (PLEG): container finished" podID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerID="cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a" exitCode=0
May 25 12:56:19.139936 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.139552    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" event={"ID":"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7","Type":"ContainerDied","Data":"cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a"}
May 25 12:56:19.141587 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.141561    2562 generic.go:358] "Generic (PLEG): container finished" podID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerID="b8db2813f5db1381780a1987b1bdc046817ca75b7e56bd51d5dc9e62e46e7f81" exitCode=0
May 25 12:56:19.141677 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.141597    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" event={"ID":"c0df651b-136c-415b-ba5f-07c28ec6d624","Type":"ContainerDied","Data":"b8db2813f5db1381780a1987b1bdc046817ca75b7e56bd51d5dc9e62e46e7f81"}
May 25 12:56:19.187177 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.187142    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:56:19.226929 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.226903    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0df651b-136c-415b-ba5f-07c28ec6d624-kserve-provision-location\") pod \"c0df651b-136c-415b-ba5f-07c28ec6d624\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") "
May 25 12:56:19.227066 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.226940    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0df651b-136c-415b-ba5f-07c28ec6d624-proxy-tls\") pod \"c0df651b-136c-415b-ba5f-07c28ec6d624\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") "
May 25 12:56:19.227066 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.227015    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkkjb\" (UniqueName: \"kubernetes.io/projected/c0df651b-136c-415b-ba5f-07c28ec6d624-kube-api-access-mkkjb\") pod \"c0df651b-136c-415b-ba5f-07c28ec6d624\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") "
May 25 12:56:19.227066 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.227056    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0df651b-136c-415b-ba5f-07c28ec6d624-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"c0df651b-136c-415b-ba5f-07c28ec6d624\" (UID: \"c0df651b-136c-415b-ba5f-07c28ec6d624\") "
May 25 12:56:19.227625 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.227251    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0df651b-136c-415b-ba5f-07c28ec6d624-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0df651b-136c-415b-ba5f-07c28ec6d624" (UID: "c0df651b-136c-415b-ba5f-07c28ec6d624"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:56:19.227714 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.227632    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0df651b-136c-415b-ba5f-07c28ec6d624-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "c0df651b-136c-415b-ba5f-07c28ec6d624" (UID: "c0df651b-136c-415b-ba5f-07c28ec6d624"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:56:19.229104 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.229075    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0df651b-136c-415b-ba5f-07c28ec6d624-kube-api-access-mkkjb" (OuterVolumeSpecName: "kube-api-access-mkkjb") pod "c0df651b-136c-415b-ba5f-07c28ec6d624" (UID: "c0df651b-136c-415b-ba5f-07c28ec6d624"). InnerVolumeSpecName "kube-api-access-mkkjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:56:19.229193 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.229136    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0df651b-136c-415b-ba5f-07c28ec6d624-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0df651b-136c-415b-ba5f-07c28ec6d624" (UID: "c0df651b-136c-415b-ba5f-07c28ec6d624"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:56:19.327892 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.327812    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkkjb\" (UniqueName: \"kubernetes.io/projected/c0df651b-136c-415b-ba5f-07c28ec6d624-kube-api-access-mkkjb\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:56:19.327892 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.327856    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0df651b-136c-415b-ba5f-07c28ec6d624-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:56:19.327892 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.327873    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0df651b-136c-415b-ba5f-07c28ec6d624-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:56:19.327892 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:19.327887    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0df651b-136c-415b-ba5f-07c28ec6d624-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:56:20.146801 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.146765    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl" event={"ID":"c0df651b-136c-415b-ba5f-07c28ec6d624","Type":"ContainerDied","Data":"29661e4750e1417fcbd7c29d492699fbcde07dfe29c866206cacd02a0579bbcf"}
May 25 12:56:20.146801 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.146802    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"
May 25 12:56:20.147316 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.146818    2562 scope.go:117] "RemoveContainer" containerID="9f92150a129efa6e2cb53d251371d140559fef16bebfbf1a85dffb8fe93fd7ed"
May 25 12:56:20.148662 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.148639    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" event={"ID":"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7","Type":"ContainerStarted","Data":"80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740"}
May 25 12:56:20.148773 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.148670    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" event={"ID":"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7","Type":"ContainerStarted","Data":"34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e"}
May 25 12:56:20.148891 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.148875    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:20.148962 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.148938    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:20.154832 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.154815    2562 scope.go:117] "RemoveContainer" containerID="b8db2813f5db1381780a1987b1bdc046817ca75b7e56bd51d5dc9e62e46e7f81"
May 25 12:56:20.161490 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.161470    2562 scope.go:117] "RemoveContainer" containerID="eebcd86cb5ee46028d1d30d7ed991c9e021635e9c273d3dd3405f31e662248a4"
May 25 12:56:20.169895 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.169854    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" podStartSLOduration=6.169842301 podStartE2EDuration="6.169842301s" podCreationTimestamp="2026-05-25 12:56:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:56:20.168411298 +0000 UTC m=+2615.826742733" watchObservedRunningTime="2026-05-25 12:56:20.169842301 +0000 UTC m=+2615.828173705"
May 25 12:56:20.179839 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.179818    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"]
May 25 12:56:20.183630 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.183611    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wggxl"]
May 25 12:56:20.957182 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:20.957150    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" path="/var/lib/kubelet/pods/c0df651b-136c-415b-ba5f-07c28ec6d624/volumes"
May 25 12:56:26.157769 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:26.157739    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:56:56.158469 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:56:56.158427    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 12:57:06.158989 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:06.158951    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 12:57:16.158893 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:16.158857    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 12:57:26.159105 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:26.159066    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 12:57:36.161788 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:36.161694    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:57:44.676241 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:44.676204    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"]
May 25 12:57:44.676716 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:44.676543    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kserve-container" containerID="cri-o://34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e" gracePeriod=30
May 25 12:57:44.676716 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:44.676593    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kube-rbac-proxy" containerID="cri-o://80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740" gracePeriod=30
May 25 12:57:45.377539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:45.377502    2562 generic.go:358] "Generic (PLEG): container finished" podID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerID="80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740" exitCode=2
May 25 12:57:45.377539 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:45.377549    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" event={"ID":"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7","Type":"ContainerDied","Data":"80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740"}
May 25 12:57:46.153856 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:46.153814    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
May 25 12:57:46.158397 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:46.158368    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
May 25 12:57:49.910219 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:49.910188    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:57:49.992651 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:49.992621    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") "
May 25 12:57:49.992819 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:49.992672    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7kgl\" (UniqueName: \"kubernetes.io/projected/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kube-api-access-w7kgl\") pod \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") "
May 25 12:57:49.992819 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:49.992716    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-proxy-tls\") pod \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") "
May 25 12:57:49.992819 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:49.992775    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kserve-provision-location\") pod \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\" (UID: \"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7\") "
May 25 12:57:49.993124 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:49.993098    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" (UID: "b24a6a56-ac9a-4654-ae98-10ac72eb6bf7"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:57:49.993196 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:49.993166    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" (UID: "b24a6a56-ac9a-4654-ae98-10ac72eb6bf7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:57:49.994926 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:49.994889    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" (UID: "b24a6a56-ac9a-4654-ae98-10ac72eb6bf7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:57:49.994926 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:49.994896    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kube-api-access-w7kgl" (OuterVolumeSpecName: "kube-api-access-w7kgl") pod "b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" (UID: "b24a6a56-ac9a-4654-ae98-10ac72eb6bf7"). InnerVolumeSpecName "kube-api-access-w7kgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:57:50.094209 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.094129    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:57:50.094209 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.094161    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:57:50.094209 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.094173    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:57:50.094209 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.094184    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7kgl\" (UniqueName: \"kubernetes.io/projected/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7-kube-api-access-w7kgl\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 12:57:50.392205 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.392103    2562 generic.go:358] "Generic (PLEG): container finished" podID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerID="34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e" exitCode=0
May 25 12:57:50.392205 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.392161    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" event={"ID":"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7","Type":"ContainerDied","Data":"34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e"}
May 25 12:57:50.392205 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.392182    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"
May 25 12:57:50.392205 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.392188    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv" event={"ID":"b24a6a56-ac9a-4654-ae98-10ac72eb6bf7","Type":"ContainerDied","Data":"25b9c67a9414273ae8bc2bc0e5b548500be082d7bd5c688a708f21bcce888c20"}
May 25 12:57:50.392529 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.392217    2562 scope.go:117] "RemoveContainer" containerID="80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740"
May 25 12:57:50.400546 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.400090    2562 scope.go:117] "RemoveContainer" containerID="34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e"
May 25 12:57:50.408236 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.408211    2562 scope.go:117] "RemoveContainer" containerID="cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a"
May 25 12:57:50.415567 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.415546    2562 scope.go:117] "RemoveContainer" containerID="80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740"
May 25 12:57:50.415829 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:57:50.415809    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740\": container with ID starting with 80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740 not found: ID does not exist" containerID="80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740"
May 25 12:57:50.415875 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.415837    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740"} err="failed to get container status \"80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740\": rpc error: code = NotFound desc = could not find container \"80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740\": container with ID starting with 80e52ceeecd2719e9f11a58e18517a993f3ba1e99e270a5029df99d33ac2a740 not found: ID does not exist"
May 25 12:57:50.415875 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.415856    2562 scope.go:117] "RemoveContainer" containerID="34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e"
May 25 12:57:50.416097 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:57:50.416079    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e\": container with ID starting with 34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e not found: ID does not exist" containerID="34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e"
May 25 12:57:50.416137 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.416104    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e"} err="failed to get container status \"34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e\": rpc error: code = NotFound desc = could not find container \"34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e\": container with ID starting with 34e5b74081a8f2edeadb19fa393a59e37377fcd4346a2556cf4b67f00d85646e not found: ID does not exist"
May 25 12:57:50.416137 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.416119    2562 scope.go:117] "RemoveContainer" containerID="cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a"
May 25 12:57:50.416341 ip-10-0-141-91 kubenswrapper[2562]: E0525 12:57:50.416322    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a\": container with ID starting with cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a not found: ID does not exist" containerID="cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a"
May 25 12:57:50.416404 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.416343    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a"} err="failed to get container status \"cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a\": rpc error: code = NotFound desc = could not find container \"cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a\": container with ID starting with cd44adaed1666ffda9fd7237292e21e68b98f8c5ad12a42f339d8709dda2847a not found: ID does not exist"
May 25 12:57:50.418974 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.418953    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"]
May 25 12:57:50.422876 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.422842    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-s8zcv"]
May 25 12:57:50.954583 ip-10-0-141-91 kubenswrapper[2562]: I0525 12:57:50.954545    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" path="/var/lib/kubelet/pods/b24a6a56-ac9a-4654-ae98-10ac72eb6bf7/volumes"
May 25 13:04:25.157250 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157213    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"]
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157471    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="storage-initializer"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157482    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="storage-initializer"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157492    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kserve-container"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157498    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kserve-container"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157507    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kube-rbac-proxy"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157512    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kube-rbac-proxy"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157518    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="storage-initializer"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157523    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="storage-initializer"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157530    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kserve-container"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157535    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kserve-container"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157543    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kube-rbac-proxy"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157548    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kube-rbac-proxy"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157591    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kserve-container"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157599    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kserve-container"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157606    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="c0df651b-136c-415b-ba5f-07c28ec6d624" containerName="kube-rbac-proxy"
May 25 13:04:25.157737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.157613    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b24a6a56-ac9a-4654-ae98-10ac72eb6bf7" containerName="kube-rbac-proxy"
May 25 13:04:25.160435 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.160416    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.163081 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.163056    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
May 25 13:04:25.163209 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.163079    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
May 25 13:04:25.163209 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.163081    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:04:25.163209 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.163063    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:04:25.163209 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.163197    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 13:04:25.167700 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.167563    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"]
May 25 13:04:25.213988 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.213946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrghw\" (UniqueName: \"kubernetes.io/projected/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kube-api-access-lrghw\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.214152 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.213995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.214152 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.214021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.214152 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.214049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.315225 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.315188    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.315441 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.315243    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lrghw\" (UniqueName: \"kubernetes.io/projected/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kube-api-access-lrghw\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.315441 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:04:25.315369    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
May 25 13:04:25.315441 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:04:25.315432    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-proxy-tls podName:7f7d5c39-0ce5-4ee5-9561-dcb375baa216 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:25.815415368 +0000 UTC m=+3101.473746752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-proxy-tls") pod "isvc-tensorflow-predictor-6756f669d7-lvzzh" (UID: "7f7d5c39-0ce5-4ee5-9561-dcb375baa216") : secret "isvc-tensorflow-predictor-serving-cert" not found
May 25 13:04:25.315620 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.315451    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.315620 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.315480    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.315855 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.315838    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.316091 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.316074    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.323558 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.323536    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrghw\" (UniqueName: \"kubernetes.io/projected/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kube-api-access-lrghw\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.819427 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.819394    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:25.821835 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:25.821815    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-lvzzh\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:26.070790 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:26.070704    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:26.192447 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:26.192416    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"]
May 25 13:04:26.195407 ip-10-0-141-91 kubenswrapper[2562]: W0525 13:04:26.195373    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7d5c39_0ce5_4ee5_9561_dcb375baa216.slice/crio-1599afd73f464a2f6503c994ab139497a3a3248d92f0c1f546c99283fd360194 WatchSource:0}: Error finding container 1599afd73f464a2f6503c994ab139497a3a3248d92f0c1f546c99283fd360194: Status 404 returned error can't find the container with id 1599afd73f464a2f6503c994ab139497a3a3248d92f0c1f546c99283fd360194
May 25 13:04:26.197226 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:26.197210    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:04:26.436324 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:26.436231    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" event={"ID":"7f7d5c39-0ce5-4ee5-9561-dcb375baa216","Type":"ContainerStarted","Data":"a188070229cfe3d0627c55603e6331541e8c402da5038682f0f7a7caaa285b73"}
May 25 13:04:26.436324 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:26.436291    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" event={"ID":"7f7d5c39-0ce5-4ee5-9561-dcb375baa216","Type":"ContainerStarted","Data":"1599afd73f464a2f6503c994ab139497a3a3248d92f0c1f546c99283fd360194"}
May 25 13:04:31.449205 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:31.449168    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerID="a188070229cfe3d0627c55603e6331541e8c402da5038682f0f7a7caaa285b73" exitCode=0
May 25 13:04:31.449205 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:31.449208    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" event={"ID":"7f7d5c39-0ce5-4ee5-9561-dcb375baa216","Type":"ContainerDied","Data":"a188070229cfe3d0627c55603e6331541e8c402da5038682f0f7a7caaa285b73"}
May 25 13:04:35.460516 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:35.460482    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" event={"ID":"7f7d5c39-0ce5-4ee5-9561-dcb375baa216","Type":"ContainerStarted","Data":"8c9cb69f74088416f58b869a0bda7ef966a1e94823caa3135364927ef88291d0"}
May 25 13:04:35.460516 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:35.460523    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" event={"ID":"7f7d5c39-0ce5-4ee5-9561-dcb375baa216","Type":"ContainerStarted","Data":"3670d18a306899c4480ffeca94b81642b0023ad5f991ed3e4496c3bb3af18aec"}
May 25 13:04:35.460931 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:35.460721    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:35.478413 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:35.478351    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podStartSLOduration=6.618891897 podStartE2EDuration="10.478331664s" podCreationTimestamp="2026-05-25 13:04:25 +0000 UTC" firstStartedPulling="2026-05-25 13:04:31.450353189 +0000 UTC m=+3107.108684576" lastFinishedPulling="2026-05-25 13:04:35.309792943 +0000 UTC m=+3110.968124343" observedRunningTime="2026-05-25 13:04:35.476380217 +0000 UTC m=+3111.134711621" watchObservedRunningTime="2026-05-25 13:04:35.478331664 +0000 UTC m=+3111.136663071"
May 25 13:04:36.463504 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:36.463472    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:36.464782 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:36.464754    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 25 13:04:37.466224 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:37.466181    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 25 13:04:42.470939 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:42.470910    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:04:42.471568 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:42.471540    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 25 13:04:52.472640 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:04:52.472609    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:05:06.143335 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:06.143231    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"]
May 25 13:05:06.143948 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:06.143705    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy" containerID="cri-o://8c9cb69f74088416f58b869a0bda7ef966a1e94823caa3135364927ef88291d0" gracePeriod=30
May 25 13:05:06.143948 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:06.143673    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kserve-container" containerID="cri-o://3670d18a306899c4480ffeca94b81642b0023ad5f991ed3e4496c3bb3af18aec" gracePeriod=30
May 25 13:05:06.547101 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:06.547065    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerID="8c9cb69f74088416f58b869a0bda7ef966a1e94823caa3135364927ef88291d0" exitCode=2
May 25 13:05:06.547287 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:06.547130    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" event={"ID":"7f7d5c39-0ce5-4ee5-9561-dcb375baa216","Type":"ContainerDied","Data":"8c9cb69f74088416f58b869a0bda7ef966a1e94823caa3135364927ef88291d0"}
May 25 13:05:07.467243 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:07.467183    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:05:12.466722 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:12.466681    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:05:17.466734 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:17.466688    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:05:17.467112 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:17.466817    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:05:22.467114 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:22.467064    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:05:27.466517 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:27.466463    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:05:32.466555 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:32.466512    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:05:36.627276 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.627229    2562 generic.go:358] "Generic (PLEG): container finished" podID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerID="3670d18a306899c4480ffeca94b81642b0023ad5f991ed3e4496c3bb3af18aec" exitCode=137
May 25 13:05:36.627647 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.627302    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" event={"ID":"7f7d5c39-0ce5-4ee5-9561-dcb375baa216","Type":"ContainerDied","Data":"3670d18a306899c4480ffeca94b81642b0023ad5f991ed3e4496c3bb3af18aec"}
May 25 13:05:36.776392 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.776369    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:05:36.842599 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.842566    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kserve-provision-location\") pod \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") "
May 25 13:05:36.842858 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.842615    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrghw\" (UniqueName: \"kubernetes.io/projected/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kube-api-access-lrghw\") pod \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") "
May 25 13:05:36.842858 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.842636    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-proxy-tls\") pod \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") "
May 25 13:05:36.842858 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.842663    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\" (UID: \"7f7d5c39-0ce5-4ee5-9561-dcb375baa216\") "
May 25 13:05:36.843044 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.842989    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "7f7d5c39-0ce5-4ee5-9561-dcb375baa216" (UID: "7f7d5c39-0ce5-4ee5-9561-dcb375baa216"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:05:36.844775 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.844737    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kube-api-access-lrghw" (OuterVolumeSpecName: "kube-api-access-lrghw") pod "7f7d5c39-0ce5-4ee5-9561-dcb375baa216" (UID: "7f7d5c39-0ce5-4ee5-9561-dcb375baa216"). InnerVolumeSpecName "kube-api-access-lrghw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:05:36.844775 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.844737    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f7d5c39-0ce5-4ee5-9561-dcb375baa216" (UID: "7f7d5c39-0ce5-4ee5-9561-dcb375baa216"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:05:36.853357 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.853330    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f7d5c39-0ce5-4ee5-9561-dcb375baa216" (UID: "7f7d5c39-0ce5-4ee5-9561-dcb375baa216"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:05:36.943217 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.943148    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:05:36.943217 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.943176    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrghw\" (UniqueName: \"kubernetes.io/projected/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-kube-api-access-lrghw\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:05:36.943217 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.943186    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:05:36.943217 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:36.943198    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7d5c39-0ce5-4ee5-9561-dcb375baa216-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:05:37.631623 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:37.631596    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"
May 25 13:05:37.632048 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:37.631588    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh" event={"ID":"7f7d5c39-0ce5-4ee5-9561-dcb375baa216","Type":"ContainerDied","Data":"1599afd73f464a2f6503c994ab139497a3a3248d92f0c1f546c99283fd360194"}
May 25 13:05:37.632048 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:37.631720    2562 scope.go:117] "RemoveContainer" containerID="8c9cb69f74088416f58b869a0bda7ef966a1e94823caa3135364927ef88291d0"
May 25 13:05:37.639322 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:37.639299    2562 scope.go:117] "RemoveContainer" containerID="3670d18a306899c4480ffeca94b81642b0023ad5f991ed3e4496c3bb3af18aec"
May 25 13:05:37.646125 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:37.646105    2562 scope.go:117] "RemoveContainer" containerID="a188070229cfe3d0627c55603e6331541e8c402da5038682f0f7a7caaa285b73"
May 25 13:05:37.666683 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:37.666661    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"]
May 25 13:05:37.670882 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:37.670860    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-lvzzh"]
May 25 13:05:38.954484 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:38.954444    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" path="/var/lib/kubelet/pods/7f7d5c39-0ce5-4ee5-9561-dcb375baa216/volumes"
May 25 13:05:47.472152 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.472112    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"]
May 25 13:05:47.472642 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.472481    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy"
May 25 13:05:47.472642 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.472499    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy"
May 25 13:05:47.472642 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.472521    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kserve-container"
May 25 13:05:47.472642 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.472529    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kserve-container"
May 25 13:05:47.472642 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.472539    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="storage-initializer"
May 25 13:05:47.472642 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.472547    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="storage-initializer"
May 25 13:05:47.472642 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.472614    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kserve-container"
May 25 13:05:47.472642 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.472625    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="7f7d5c39-0ce5-4ee5-9561-dcb375baa216" containerName="kube-rbac-proxy"
May 25 13:05:47.475396 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.475375    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.478731 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.478701    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
May 25 13:05:47.479157 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.479140    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:05:47.479457 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.479435    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:05:47.479912 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.479895    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 13:05:47.480275 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.480241    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
May 25 13:05:47.485825 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.485798    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"]
May 25 13:05:47.521908 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.521865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.521908 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.521915    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.522139 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.521970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.522139 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.522002    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cph8x\" (UniqueName: \"kubernetes.io/projected/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kube-api-access-cph8x\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.622391 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.622357    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cph8x\" (UniqueName: \"kubernetes.io/projected/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kube-api-access-cph8x\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.622567 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.622396    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.622567 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.622439    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.622567 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.622489    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.622831 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.622810    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.623155 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.623129    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.624824 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.624807    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.630617 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.630595    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cph8x\" (UniqueName: \"kubernetes.io/projected/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kube-api-access-cph8x\") pod \"isvc-triton-predictor-84bb65d94b-87wbf\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.789062 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.789026    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:05:47.913572 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:47.913547    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"]
May 25 13:05:47.916170 ip-10-0-141-91 kubenswrapper[2562]: W0525 13:05:47.916139    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bd29dd_8ffb_4d68_a2fa_007b1b5b0549.slice/crio-d254c2eda83b467c9105199b847cea3a1679b87af44d84cee29edeee6490f551 WatchSource:0}: Error finding container d254c2eda83b467c9105199b847cea3a1679b87af44d84cee29edeee6490f551: Status 404 returned error can't find the container with id d254c2eda83b467c9105199b847cea3a1679b87af44d84cee29edeee6490f551
May 25 13:05:48.665763 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:48.665726    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" event={"ID":"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549","Type":"ContainerStarted","Data":"de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f"}
May 25 13:05:48.665763 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:48.665768    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" event={"ID":"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549","Type":"ContainerStarted","Data":"d254c2eda83b467c9105199b847cea3a1679b87af44d84cee29edeee6490f551"}
May 25 13:05:51.675465 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:51.675431    2562 generic.go:358] "Generic (PLEG): container finished" podID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerID="de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f" exitCode=0
May 25 13:05:51.675843 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:05:51.675479    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" event={"ID":"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549","Type":"ContainerDied","Data":"de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f"}
May 25 13:07:46.041324 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:46.041289    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" event={"ID":"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549","Type":"ContainerStarted","Data":"432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186"}
May 25 13:07:46.041324 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:46.041327    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" event={"ID":"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549","Type":"ContainerStarted","Data":"83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b"}
May 25 13:07:46.041817 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:46.041417    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:07:46.060883 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:46.060834    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" podStartSLOduration=5.682166807 podStartE2EDuration="1m59.060819437s" podCreationTimestamp="2026-05-25 13:05:47 +0000 UTC" firstStartedPulling="2026-05-25 13:05:51.676494999 +0000 UTC m=+3187.334826382" lastFinishedPulling="2026-05-25 13:07:45.055147627 +0000 UTC m=+3300.713479012" observedRunningTime="2026-05-25 13:07:46.059572666 +0000 UTC m=+3301.717904070" watchObservedRunningTime="2026-05-25 13:07:46.060819437 +0000 UTC m=+3301.719150842"
May 25 13:07:47.043763 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:47.043726    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:07:47.044820 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:47.044791    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:07:48.046461 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:48.046426    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:07:53.052039 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:53.052006    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:07:53.052879 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:53.052858    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:07:58.988245 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:58.988207    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"]
May 25 13:07:58.988770 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:58.988511    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kserve-container" containerID="cri-o://83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b" gracePeriod=30
May 25 13:07:58.988770 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:07:58.988587    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kube-rbac-proxy" containerID="cri-o://432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186" gracePeriod=30
May 25 13:08:00.076868 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:00.076833    2562 generic.go:358] "Generic (PLEG): container finished" podID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerID="432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186" exitCode=2
May 25 13:08:00.077339 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:00.076902    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" event={"ID":"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549","Type":"ContainerDied","Data":"432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186"}
May 25 13:08:01.929908 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:01.929886    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:08:02.038666 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.038634    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kserve-provision-location\") pod \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") "
May 25 13:08:02.038910 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.038677    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-proxy-tls\") pod \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") "
May 25 13:08:02.038910 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.038734    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cph8x\" (UniqueName: \"kubernetes.io/projected/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kube-api-access-cph8x\") pod \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") "
May 25 13:08:02.038910 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.038758    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-isvc-triton-kube-rbac-proxy-sar-config\") pod \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\" (UID: \"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549\") "
May 25 13:08:02.039092 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.039073    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" (UID: "b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:02.039151 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.039113    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" (UID: "b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:08:02.040814 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.040794    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" (UID: "b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:02.040907 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.040884    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kube-api-access-cph8x" (OuterVolumeSpecName: "kube-api-access-cph8x") pod "b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" (UID: "b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549"). InnerVolumeSpecName "kube-api-access-cph8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:08:02.083814 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.083777    2562 generic.go:358] "Generic (PLEG): container finished" podID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerID="83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b" exitCode=0
May 25 13:08:02.083958 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.083823    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" event={"ID":"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549","Type":"ContainerDied","Data":"83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b"}
May 25 13:08:02.083958 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.083848    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf" event={"ID":"b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549","Type":"ContainerDied","Data":"d254c2eda83b467c9105199b847cea3a1679b87af44d84cee29edeee6490f551"}
May 25 13:08:02.083958 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.083865    2562 scope.go:117] "RemoveContainer" containerID="432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186"
May 25 13:08:02.083958 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.083868    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"
May 25 13:08:02.091229 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.091090    2562 scope.go:117] "RemoveContainer" containerID="83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b"
May 25 13:08:02.097903 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.097886    2562 scope.go:117] "RemoveContainer" containerID="de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f"
May 25 13:08:02.104247 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.104225    2562 scope.go:117] "RemoveContainer" containerID="432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186"
May 25 13:08:02.104682 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:08:02.104594    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186\": container with ID starting with 432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186 not found: ID does not exist" containerID="432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186"
May 25 13:08:02.104682 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.104627    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186"} err="failed to get container status \"432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186\": rpc error: code = NotFound desc = could not find container \"432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186\": container with ID starting with 432d083100cf7fdeb417356a66a43d15c1d3faca30ca8fdb976e5e639c51e186 not found: ID does not exist"
May 25 13:08:02.104682 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.104664    2562 scope.go:117] "RemoveContainer" containerID="83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b"
May 25 13:08:02.104939 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:08:02.104920    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b\": container with ID starting with 83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b not found: ID does not exist" containerID="83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b"
May 25 13:08:02.104996 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.104943    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b"} err="failed to get container status \"83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b\": rpc error: code = NotFound desc = could not find container \"83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b\": container with ID starting with 83eb659de2f97bcefeb0c1b9489a43ba54ae67bdacc893bd82da1e616a876b2b not found: ID does not exist"
May 25 13:08:02.104996 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.104959    2562 scope.go:117] "RemoveContainer" containerID="de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f"
May 25 13:08:02.105183 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:08:02.105161    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f\": container with ID starting with de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f not found: ID does not exist" containerID="de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f"
May 25 13:08:02.105240 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.105189    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f"} err="failed to get container status \"de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f\": rpc error: code = NotFound desc = could not find container \"de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f\": container with ID starting with de5218cb4716174676cde9851986391d569e9781c3e462744e7a545ffccf209f not found: ID does not exist"
May 25 13:08:02.105842 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.105819    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"]
May 25 13:08:02.110797 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.110778    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-87wbf"]
May 25 13:08:02.139862 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.139830    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cph8x\" (UniqueName: \"kubernetes.io/projected/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kube-api-access-cph8x\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:08:02.139862 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.139858    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:08:02.140012 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.139869    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:08:02.140012 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.139880    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:08:02.954829 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:08:02.954796    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" path="/var/lib/kubelet/pods/b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549/volumes"
May 25 13:11:19.632023 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.631928    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"]
May 25 13:11:19.632505 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.632324    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kserve-container"
May 25 13:11:19.632505 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.632340    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kserve-container"
May 25 13:11:19.632505 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.632356    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kube-rbac-proxy"
May 25 13:11:19.632505 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.632364    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kube-rbac-proxy"
May 25 13:11:19.632505 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.632391    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="storage-initializer"
May 25 13:11:19.632505 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.632400    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="storage-initializer"
May 25 13:11:19.632505 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.632456    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kube-rbac-proxy"
May 25 13:11:19.632505 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.632468    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b4bd29dd-8ffb-4d68-a2fa-007b1b5b0549" containerName="kserve-container"
May 25 13:11:19.634503 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.634484    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.637023 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.636996    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
May 25 13:11:19.637153 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.637089    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:11:19.637153 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.637089    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
May 25 13:11:19.637153 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.637132    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:11:19.637153 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.637147    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 13:11:19.646180 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.646157    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"]
May 25 13:11:19.745714 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.745680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10b02418-072a-41ca-a345-b1def6a887f5-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.745714 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.745719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plg46\" (UniqueName: \"kubernetes.io/projected/10b02418-072a-41ca-a345-b1def6a887f5-kube-api-access-plg46\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.745953 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.745799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10b02418-072a-41ca-a345-b1def6a887f5-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.745953 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.745848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10b02418-072a-41ca-a345-b1def6a887f5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.846360 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.846316    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10b02418-072a-41ca-a345-b1def6a887f5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.846569 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.846376    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10b02418-072a-41ca-a345-b1def6a887f5-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.846569 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.846399    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-plg46\" (UniqueName: \"kubernetes.io/projected/10b02418-072a-41ca-a345-b1def6a887f5-kube-api-access-plg46\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.846569 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.846432    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10b02418-072a-41ca-a345-b1def6a887f5-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.846569 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:11:19.846494    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
May 25 13:11:19.846787 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:11:19.846586    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b02418-072a-41ca-a345-b1def6a887f5-proxy-tls podName:10b02418-072a-41ca-a345-b1def6a887f5 nodeName:}" failed. No retries permitted until 2026-05-25 13:11:20.346564155 +0000 UTC m=+3516.004895553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/10b02418-072a-41ca-a345-b1def6a887f5-proxy-tls") pod "isvc-xgboost-runtime-predictor-779db84d9-gjlvb" (UID: "10b02418-072a-41ca-a345-b1def6a887f5") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
May 25 13:11:19.846787 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.846766    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10b02418-072a-41ca-a345-b1def6a887f5-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.847064 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.847045    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10b02418-072a-41ca-a345-b1def6a887f5-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:19.854848 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:19.854827    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plg46\" (UniqueName: \"kubernetes.io/projected/10b02418-072a-41ca-a345-b1def6a887f5-kube-api-access-plg46\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:20.350189 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:20.350138    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10b02418-072a-41ca-a345-b1def6a887f5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:20.352616 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:20.352592    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10b02418-072a-41ca-a345-b1def6a887f5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-gjlvb\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:20.544212 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:20.544177    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:20.664340 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:20.664308    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"]
May 25 13:11:20.667383 ip-10-0-141-91 kubenswrapper[2562]: W0525 13:11:20.667354    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b02418_072a_41ca_a345_b1def6a887f5.slice/crio-8d41217aeb73362c46c23fe0cd714ee677c3826cd6ae963d84fdd4c4e456cbc5 WatchSource:0}: Error finding container 8d41217aeb73362c46c23fe0cd714ee677c3826cd6ae963d84fdd4c4e456cbc5: Status 404 returned error can't find the container with id 8d41217aeb73362c46c23fe0cd714ee677c3826cd6ae963d84fdd4c4e456cbc5
May 25 13:11:20.669142 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:20.669126    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:11:21.596293 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:21.596233    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" event={"ID":"10b02418-072a-41ca-a345-b1def6a887f5","Type":"ContainerStarted","Data":"ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27"}
May 25 13:11:21.596462 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:21.596295    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" event={"ID":"10b02418-072a-41ca-a345-b1def6a887f5","Type":"ContainerStarted","Data":"8d41217aeb73362c46c23fe0cd714ee677c3826cd6ae963d84fdd4c4e456cbc5"}
May 25 13:11:24.606329 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:24.606296    2562 generic.go:358] "Generic (PLEG): container finished" podID="10b02418-072a-41ca-a345-b1def6a887f5" containerID="ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27" exitCode=0
May 25 13:11:24.606620 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:24.606341    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" event={"ID":"10b02418-072a-41ca-a345-b1def6a887f5","Type":"ContainerDied","Data":"ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27"}
May 25 13:11:42.661417 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:42.661332    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" event={"ID":"10b02418-072a-41ca-a345-b1def6a887f5","Type":"ContainerStarted","Data":"5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c"}
May 25 13:11:42.661417 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:42.661373    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" event={"ID":"10b02418-072a-41ca-a345-b1def6a887f5","Type":"ContainerStarted","Data":"f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134"}
May 25 13:11:42.661850 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:42.661572    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:42.680955 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:42.680908    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podStartSLOduration=5.993485086 podStartE2EDuration="23.680894565s" podCreationTimestamp="2026-05-25 13:11:19 +0000 UTC" firstStartedPulling="2026-05-25 13:11:24.60746308 +0000 UTC m=+3520.265794463" lastFinishedPulling="2026-05-25 13:11:42.294872546 +0000 UTC m=+3537.953203942" observedRunningTime="2026-05-25 13:11:42.678856294 +0000 UTC m=+3538.337187699" watchObservedRunningTime="2026-05-25 13:11:42.680894565 +0000 UTC m=+3538.339225969"
May 25 13:11:43.664165 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:43.664131    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:43.665192 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:43.665166    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:11:44.666790 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:44.666748    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:11:49.671172 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:49.671141    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:11:49.671760 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:49.671733    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:11:59.671803 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:11:59.671757    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:12:09.671828 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:12:09.671786    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:12:19.672421 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:12:19.672384    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:12:29.672252 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:12:29.672214    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:12:39.672664 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:12:39.672569    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:12:49.672387 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:12:49.672355    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:12:59.783805 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:12:59.783763    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"]
May 25 13:12:59.784225 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:12:59.784174    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container" containerID="cri-o://f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134" gracePeriod=30
May 25 13:12:59.784291 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:12:59.784230    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kube-rbac-proxy" containerID="cri-o://5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c" gracePeriod=30
May 25 13:13:00.870196 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:00.870159    2562 generic.go:358] "Generic (PLEG): container finished" podID="10b02418-072a-41ca-a345-b1def6a887f5" containerID="5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c" exitCode=2
May 25 13:13:00.870590 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:00.870230    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" event={"ID":"10b02418-072a-41ca-a345-b1def6a887f5","Type":"ContainerDied","Data":"5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c"}
May 25 13:13:03.229473 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.229451    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:13:03.322763 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.322677    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10b02418-072a-41ca-a345-b1def6a887f5-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"10b02418-072a-41ca-a345-b1def6a887f5\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") "
May 25 13:13:03.322763 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.322736    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10b02418-072a-41ca-a345-b1def6a887f5-proxy-tls\") pod \"10b02418-072a-41ca-a345-b1def6a887f5\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") "
May 25 13:13:03.322952 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.322848    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plg46\" (UniqueName: \"kubernetes.io/projected/10b02418-072a-41ca-a345-b1def6a887f5-kube-api-access-plg46\") pod \"10b02418-072a-41ca-a345-b1def6a887f5\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") "
May 25 13:13:03.322952 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.322889    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10b02418-072a-41ca-a345-b1def6a887f5-kserve-provision-location\") pod \"10b02418-072a-41ca-a345-b1def6a887f5\" (UID: \"10b02418-072a-41ca-a345-b1def6a887f5\") "
May 25 13:13:03.323101 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.323077    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b02418-072a-41ca-a345-b1def6a887f5-isvc-xgboost-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-runtime-kube-rbac-proxy-sar-config") pod "10b02418-072a-41ca-a345-b1def6a887f5" (UID: "10b02418-072a-41ca-a345-b1def6a887f5"). InnerVolumeSpecName "isvc-xgboost-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:13:03.323246 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.323222    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b02418-072a-41ca-a345-b1def6a887f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10b02418-072a-41ca-a345-b1def6a887f5" (UID: "10b02418-072a-41ca-a345-b1def6a887f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:13:03.324886 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.324860    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b02418-072a-41ca-a345-b1def6a887f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10b02418-072a-41ca-a345-b1def6a887f5" (UID: "10b02418-072a-41ca-a345-b1def6a887f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:13:03.324886 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.324865    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b02418-072a-41ca-a345-b1def6a887f5-kube-api-access-plg46" (OuterVolumeSpecName: "kube-api-access-plg46") pod "10b02418-072a-41ca-a345-b1def6a887f5" (UID: "10b02418-072a-41ca-a345-b1def6a887f5"). InnerVolumeSpecName "kube-api-access-plg46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:13:03.423563 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.423521    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plg46\" (UniqueName: \"kubernetes.io/projected/10b02418-072a-41ca-a345-b1def6a887f5-kube-api-access-plg46\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:13:03.423563 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.423552    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10b02418-072a-41ca-a345-b1def6a887f5-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:13:03.423563 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.423564    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10b02418-072a-41ca-a345-b1def6a887f5-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:13:03.423796 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.423573    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10b02418-072a-41ca-a345-b1def6a887f5-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:13:03.879968 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.879928    2562 generic.go:358] "Generic (PLEG): container finished" podID="10b02418-072a-41ca-a345-b1def6a887f5" containerID="f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134" exitCode=0
May 25 13:13:03.880139 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.879988    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" event={"ID":"10b02418-072a-41ca-a345-b1def6a887f5","Type":"ContainerDied","Data":"f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134"}
May 25 13:13:03.880139 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.880002    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"
May 25 13:13:03.880139 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.880021    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb" event={"ID":"10b02418-072a-41ca-a345-b1def6a887f5","Type":"ContainerDied","Data":"8d41217aeb73362c46c23fe0cd714ee677c3826cd6ae963d84fdd4c4e456cbc5"}
May 25 13:13:03.880139 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.880042    2562 scope.go:117] "RemoveContainer" containerID="5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c"
May 25 13:13:03.887737 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.887714    2562 scope.go:117] "RemoveContainer" containerID="f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134"
May 25 13:13:03.894531 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.894513    2562 scope.go:117] "RemoveContainer" containerID="ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27"
May 25 13:13:03.900224 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.900202    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"]
May 25 13:13:03.901476 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.901464    2562 scope.go:117] "RemoveContainer" containerID="5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c"
May 25 13:13:03.901717 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:13:03.901699    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c\": container with ID starting with 5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c not found: ID does not exist" containerID="5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c"
May 25 13:13:03.901762 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.901727    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c"} err="failed to get container status \"5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c\": rpc error: code = NotFound desc = could not find container \"5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c\": container with ID starting with 5ced44575d851434bbf45b5551c1bf828b53032814045a27d7a48cd6a511286c not found: ID does not exist"
May 25 13:13:03.901762 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.901745    2562 scope.go:117] "RemoveContainer" containerID="f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134"
May 25 13:13:03.901954 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:13:03.901937    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134\": container with ID starting with f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134 not found: ID does not exist" containerID="f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134"
May 25 13:13:03.901994 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.901958    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134"} err="failed to get container status \"f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134\": rpc error: code = NotFound desc = could not find container \"f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134\": container with ID starting with f085aa1c6c14a1efeab30182c3ec60a6bd32ce7f1a2f9f89b1a884801cc02134 not found: ID does not exist"
May 25 13:13:03.901994 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.901971    2562 scope.go:117] "RemoveContainer" containerID="ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27"
May 25 13:13:03.902155 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:13:03.902140    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27\": container with ID starting with ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27 not found: ID does not exist" containerID="ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27"
May 25 13:13:03.902199 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.902158    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27"} err="failed to get container status \"ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27\": rpc error: code = NotFound desc = could not find container \"ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27\": container with ID starting with ec40f7cef42516a8a93e273385373ff87208ea23603b1027a0edc49176e42d27 not found: ID does not exist"
May 25 13:13:03.905253 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:03.905232    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-gjlvb"]
May 25 13:13:04.954135 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:13:04.954103    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10b02418-072a-41ca-a345-b1def6a887f5" path="/var/lib/kubelet/pods/10b02418-072a-41ca-a345-b1def6a887f5/volumes"
May 25 13:14:00.089617 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.089580    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"]
May 25 13:14:00.090010 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.089827    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kube-rbac-proxy"
May 25 13:14:00.090010 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.089838    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kube-rbac-proxy"
May 25 13:14:00.090010 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.089847    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container"
May 25 13:14:00.090010 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.089852    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container"
May 25 13:14:00.090010 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.089858    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="storage-initializer"
May 25 13:14:00.090010 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.089864    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="storage-initializer"
May 25 13:14:00.090010 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.089925    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kube-rbac-proxy"
May 25 13:14:00.090010 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.089934    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="10b02418-072a-41ca-a345-b1def6a887f5" containerName="kserve-container"
May 25 13:14:00.091860 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.091839    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.094416 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.094396    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
May 25 13:14:00.095653 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.095632    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 13:14:00.095753 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.095636    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:14:00.095824 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.095784    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:14:00.095886 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.095857    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
May 25 13:14:00.104271 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.104235    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"]
May 25 13:14:00.113071 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.113047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1763d94-b369-407c-a8e2-d383e6dd7de6-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.113157 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.113094    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdw57\" (UniqueName: \"kubernetes.io/projected/d1763d94-b369-407c-a8e2-d383e6dd7de6-kube-api-access-cdw57\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.113157 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.113121    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1763d94-b369-407c-a8e2-d383e6dd7de6-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.113157 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.113147    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1763d94-b369-407c-a8e2-d383e6dd7de6-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.214243 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.214210    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1763d94-b369-407c-a8e2-d383e6dd7de6-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.214420 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.214253    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cdw57\" (UniqueName: \"kubernetes.io/projected/d1763d94-b369-407c-a8e2-d383e6dd7de6-kube-api-access-cdw57\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.214420 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.214293    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1763d94-b369-407c-a8e2-d383e6dd7de6-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.214522 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.214468    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1763d94-b369-407c-a8e2-d383e6dd7de6-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.214654 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.214638    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1763d94-b369-407c-a8e2-d383e6dd7de6-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.215020 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.215004    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1763d94-b369-407c-a8e2-d383e6dd7de6-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.216669 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.216646    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1763d94-b369-407c-a8e2-d383e6dd7de6-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.223019 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.222997    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdw57\" (UniqueName: \"kubernetes.io/projected/d1763d94-b369-407c-a8e2-d383e6dd7de6-kube-api-access-cdw57\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.401956 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.401864    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:00.528467 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:00.528446    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"]
May 25 13:14:00.531332 ip-10-0-141-91 kubenswrapper[2562]: W0525 13:14:00.531304    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1763d94_b369_407c_a8e2_d383e6dd7de6.slice/crio-30f5b82b59b5fa4012b10b660c24185dbf807502bba2fc30c41b959f9c32edf8 WatchSource:0}: Error finding container 30f5b82b59b5fa4012b10b660c24185dbf807502bba2fc30c41b959f9c32edf8: Status 404 returned error can't find the container with id 30f5b82b59b5fa4012b10b660c24185dbf807502bba2fc30c41b959f9c32edf8
May 25 13:14:01.029447 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:01.029410    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" event={"ID":"d1763d94-b369-407c-a8e2-d383e6dd7de6","Type":"ContainerStarted","Data":"68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b"}
May 25 13:14:01.029619 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:01.029451    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" event={"ID":"d1763d94-b369-407c-a8e2-d383e6dd7de6","Type":"ContainerStarted","Data":"30f5b82b59b5fa4012b10b660c24185dbf807502bba2fc30c41b959f9c32edf8"}
May 25 13:14:05.041605 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:05.041568    2562 generic.go:358] "Generic (PLEG): container finished" podID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerID="68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b" exitCode=0
May 25 13:14:05.041974 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:05.041641    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" event={"ID":"d1763d94-b369-407c-a8e2-d383e6dd7de6","Type":"ContainerDied","Data":"68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b"}
May 25 13:14:06.046253 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:06.046218    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" event={"ID":"d1763d94-b369-407c-a8e2-d383e6dd7de6","Type":"ContainerStarted","Data":"7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11"}
May 25 13:14:06.046253 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:06.046255    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" event={"ID":"d1763d94-b369-407c-a8e2-d383e6dd7de6","Type":"ContainerStarted","Data":"52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c"}
May 25 13:14:06.046812 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:06.046486    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:06.066238 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:06.066190    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podStartSLOduration=6.066175419 podStartE2EDuration="6.066175419s" podCreationTimestamp="2026-05-25 13:14:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:14:06.065832251 +0000 UTC m=+3681.724163666" watchObservedRunningTime="2026-05-25 13:14:06.066175419 +0000 UTC m=+3681.724506828"
May 25 13:14:07.049093 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:07.049060    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:07.050120 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:07.050093    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:14:08.051819 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:08.051777    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:14:13.056771 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:13.056741    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:14:13.057373 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:13.057341    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:14:23.058193 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:23.058146    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:14:33.057639 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:33.057599    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:14:43.057838 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:43.057798    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:14:53.058303 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:14:53.058239    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:15:03.057941 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:03.057897    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:15:13.058456 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:13.058422    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:15:20.322666 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:20.322630    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"]
May 25 13:15:20.323086 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:20.322916    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" containerID="cri-o://52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c" gracePeriod=30
May 25 13:15:20.323086 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:20.322965    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kube-rbac-proxy" containerID="cri-o://7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11" gracePeriod=30
May 25 13:15:21.241229 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:21.241197    2562 generic.go:358] "Generic (PLEG): container finished" podID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerID="7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11" exitCode=2
May 25 13:15:21.241412 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:21.241279    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" event={"ID":"d1763d94-b369-407c-a8e2-d383e6dd7de6","Type":"ContainerDied","Data":"7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11"}
May 25 13:15:23.052740 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.052698    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
May 25 13:15:23.058066 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.058038    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:15:23.865624 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.865601    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:15:23.918762 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.918734    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1763d94-b369-407c-a8e2-d383e6dd7de6-kserve-provision-location\") pod \"d1763d94-b369-407c-a8e2-d383e6dd7de6\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") "
May 25 13:15:23.918937 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.918792    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdw57\" (UniqueName: \"kubernetes.io/projected/d1763d94-b369-407c-a8e2-d383e6dd7de6-kube-api-access-cdw57\") pod \"d1763d94-b369-407c-a8e2-d383e6dd7de6\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") "
May 25 13:15:23.918937 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.918823    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1763d94-b369-407c-a8e2-d383e6dd7de6-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"d1763d94-b369-407c-a8e2-d383e6dd7de6\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") "
May 25 13:15:23.918937 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.918861    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1763d94-b369-407c-a8e2-d383e6dd7de6-proxy-tls\") pod \"d1763d94-b369-407c-a8e2-d383e6dd7de6\" (UID: \"d1763d94-b369-407c-a8e2-d383e6dd7de6\") "
May 25 13:15:23.919145 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.919098    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1763d94-b369-407c-a8e2-d383e6dd7de6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1763d94-b369-407c-a8e2-d383e6dd7de6" (UID: "d1763d94-b369-407c-a8e2-d383e6dd7de6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:15:23.919321 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.919294    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1763d94-b369-407c-a8e2-d383e6dd7de6-isvc-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-kube-rbac-proxy-sar-config") pod "d1763d94-b369-407c-a8e2-d383e6dd7de6" (UID: "d1763d94-b369-407c-a8e2-d383e6dd7de6"). InnerVolumeSpecName "isvc-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:15:23.921021 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.920994    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1763d94-b369-407c-a8e2-d383e6dd7de6-kube-api-access-cdw57" (OuterVolumeSpecName: "kube-api-access-cdw57") pod "d1763d94-b369-407c-a8e2-d383e6dd7de6" (UID: "d1763d94-b369-407c-a8e2-d383e6dd7de6"). InnerVolumeSpecName "kube-api-access-cdw57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:15:23.921197 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:23.921019    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1763d94-b369-407c-a8e2-d383e6dd7de6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1763d94-b369-407c-a8e2-d383e6dd7de6" (UID: "d1763d94-b369-407c-a8e2-d383e6dd7de6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:15:24.019717 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.019675    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1763d94-b369-407c-a8e2-d383e6dd7de6-proxy-tls\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:15:24.019717 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.019717    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1763d94-b369-407c-a8e2-d383e6dd7de6-kserve-provision-location\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:15:24.019717 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.019730    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdw57\" (UniqueName: \"kubernetes.io/projected/d1763d94-b369-407c-a8e2-d383e6dd7de6-kube-api-access-cdw57\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:15:24.019965 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.019741    2562 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1763d94-b369-407c-a8e2-d383e6dd7de6-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-91.ec2.internal\" DevicePath \"\""
May 25 13:15:24.252162 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.252129    2562 generic.go:358] "Generic (PLEG): container finished" podID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerID="52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c" exitCode=0
May 25 13:15:24.252578 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.252213    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"
May 25 13:15:24.252578 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.252213    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" event={"ID":"d1763d94-b369-407c-a8e2-d383e6dd7de6","Type":"ContainerDied","Data":"52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c"}
May 25 13:15:24.252578 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.252282    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf" event={"ID":"d1763d94-b369-407c-a8e2-d383e6dd7de6","Type":"ContainerDied","Data":"30f5b82b59b5fa4012b10b660c24185dbf807502bba2fc30c41b959f9c32edf8"}
May 25 13:15:24.252578 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.252292    2562 scope.go:117] "RemoveContainer" containerID="7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11"
May 25 13:15:24.260090 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.260067    2562 scope.go:117] "RemoveContainer" containerID="52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c"
May 25 13:15:24.266642 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.266622    2562 scope.go:117] "RemoveContainer" containerID="68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b"
May 25 13:15:24.272789 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.272773    2562 scope.go:117] "RemoveContainer" containerID="7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11"
May 25 13:15:24.273011 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:15:24.272992    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11\": container with ID starting with 7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11 not found: ID does not exist" containerID="7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11"
May 25 13:15:24.273072 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.273016    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11"} err="failed to get container status \"7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11\": rpc error: code = NotFound desc = could not find container \"7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11\": container with ID starting with 7606c0e63185a039495e374984f9a12a2011c0fcaa601840cc1ffccc93315f11 not found: ID does not exist"
May 25 13:15:24.273072 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.273032    2562 scope.go:117] "RemoveContainer" containerID="52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c"
May 25 13:15:24.273333 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:15:24.273312    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c\": container with ID starting with 52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c not found: ID does not exist" containerID="52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c"
May 25 13:15:24.273398 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.273340    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c"} err="failed to get container status \"52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c\": rpc error: code = NotFound desc = could not find container \"52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c\": container with ID starting with 52c21232f309f1e98db7c656b1b2385ae541ece5af5cea0003c9f33c1b8d586c not found: ID does not exist"
May 25 13:15:24.273398 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.273357    2562 scope.go:117] "RemoveContainer" containerID="68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b"
May 25 13:15:24.273603 ip-10-0-141-91 kubenswrapper[2562]: E0525 13:15:24.273588    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b\": container with ID starting with 68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b not found: ID does not exist" containerID="68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b"
May 25 13:15:24.273657 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.273606    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b"} err="failed to get container status \"68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b\": rpc error: code = NotFound desc = could not find container \"68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b\": container with ID starting with 68634d65bca670ef4e044e347cbd4ab622040d0718419f847ad61c0426a49f0b not found: ID does not exist"
May 25 13:15:24.277546 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.277524    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"]
May 25 13:15:24.283307 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.283286    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-z6zhf"]
May 25 13:15:24.954576 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:15:24.954546    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" path="/var/lib/kubelet/pods/d1763d94-b369-407c-a8e2-d383e6dd7de6/volumes"
May 25 13:21:42.403624 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:42.403526    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wlsj7_3f71c2f0-743b-4295-af1d-2182812eaeb6/global-pull-secret-syncer/0.log"
May 25 13:21:42.567237 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:42.567186    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dg8fc_128d93e5-705b-4d8f-b4b0-60af82c40726/konnectivity-agent/0.log"
May 25 13:21:42.696089 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:42.695996    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-91.ec2.internal_8e4a54b448df6387855e6ddf700b60d5/haproxy/0.log"
May 25 13:21:46.632963 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:46.632925    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sh72p_c14ae715-c96a-43de-9e7b-6c9fe09ce2fd/node-exporter/0.log"
May 25 13:21:46.654748 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:46.654715    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sh72p_c14ae715-c96a-43de-9e7b-6c9fe09ce2fd/kube-rbac-proxy/0.log"
May 25 13:21:46.678975 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:46.678947    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sh72p_c14ae715-c96a-43de-9e7b-6c9fe09ce2fd/init-textfile/0.log"
May 25 13:21:48.566553 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:48.566524    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-httbc_a1ed9528-15dc-44a5-94ce-b457a0eccece/networking-console-plugin/0.log"
May 25 13:21:49.751892 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.751854    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"]
May 25 13:21:49.752297 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.752119    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kube-rbac-proxy"
May 25 13:21:49.752297 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.752135    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kube-rbac-proxy"
May 25 13:21:49.752297 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.752151    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="storage-initializer"
May 25 13:21:49.752297 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.752156    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="storage-initializer"
May 25 13:21:49.752297 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.752167    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container"
May 25 13:21:49.752297 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.752172    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container"
May 25 13:21:49.752297 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.752217    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kserve-container"
May 25 13:21:49.752297 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.752225    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="d1763d94-b369-407c-a8e2-d383e6dd7de6" containerName="kube-rbac-proxy"
May 25 13:21:49.755097 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.755074    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.757683 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.757653    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj4hj\"/\"openshift-service-ca.crt\""
May 25 13:21:49.757818 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.757759    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj4hj\"/\"kube-root-ca.crt\""
May 25 13:21:49.758829 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.758814    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wj4hj\"/\"default-dockercfg-lb72r\""
May 25 13:21:49.763653 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.763625    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"]
May 25 13:21:49.797532 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.797494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldl5k\" (UniqueName: \"kubernetes.io/projected/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-kube-api-access-ldl5k\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.797532 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.797540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-sys\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.797754 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.797560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-proc\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.797754 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.797622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-lib-modules\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.797754 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.797670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-podres\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.898621 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.898555    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldl5k\" (UniqueName: \"kubernetes.io/projected/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-kube-api-access-ldl5k\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.898819 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.898647    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-sys\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.898819 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.898679    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-proc\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.898819 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.898718    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-lib-modules\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.898819 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.898779    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-podres\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.898819 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.898801    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-proc\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.898986 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.898875    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-lib-modules\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.898986 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.898781    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-sys\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.898986 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.898916    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-podres\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:49.907023 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:49.906988    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldl5k\" (UniqueName: \"kubernetes.io/projected/98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3-kube-api-access-ldl5k\") pod \"perf-node-gather-daemonset-72zf6\" (UID: \"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:50.065166 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:50.065073    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:50.193082 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:50.193048    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"]
May 25 13:21:50.196545 ip-10-0-141-91 kubenswrapper[2562]: W0525 13:21:50.196514    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98a1dcc8_a5ba_42cd_bd90_30b6b932f3b3.slice/crio-1d2f4bcef425a9aa1cfd20cd8cbc4e33c9c02ab78cd405db1c16b99e1123cad5 WatchSource:0}: Error finding container 1d2f4bcef425a9aa1cfd20cd8cbc4e33c9c02ab78cd405db1c16b99e1123cad5: Status 404 returned error can't find the container with id 1d2f4bcef425a9aa1cfd20cd8cbc4e33c9c02ab78cd405db1c16b99e1123cad5
May 25 13:21:50.198365 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:50.198348    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:21:50.268760 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:50.268711    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6" event={"ID":"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3","Type":"ContainerStarted","Data":"1d2f4bcef425a9aa1cfd20cd8cbc4e33c9c02ab78cd405db1c16b99e1123cad5"}
May 25 13:21:50.412051 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:50.412022    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c77cg_059085b8-4425-4d8c-b229-bd2bce5c1657/dns/0.log"
May 25 13:21:50.431797 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:50.431761    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c77cg_059085b8-4425-4d8c-b229-bd2bce5c1657/kube-rbac-proxy/0.log"
May 25 13:21:50.561573 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:50.561548    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9q65_01253226-6d29-4eac-9a58-3d9a8be0032d/dns-node-resolver/0.log"
May 25 13:21:51.040666 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:51.040635    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cf9x4_cb76c49a-e523-4f5b-a0ae-3ac0d8b0442f/node-ca/0.log"
May 25 13:21:51.273415 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:51.273373    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6" event={"ID":"98a1dcc8-a5ba-42cd-bd90-30b6b932f3b3","Type":"ContainerStarted","Data":"2add1fd4ab25d90ff78f883616431923057e73daa318b71c4880d8c885c02287"}
May 25 13:21:51.273599 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:51.273498    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:21:51.290511 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:51.290452    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6" podStartSLOduration=2.290437061 podStartE2EDuration="2.290437061s" podCreationTimestamp="2026-05-25 13:21:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:21:51.290061926 +0000 UTC m=+4146.948393333" watchObservedRunningTime="2026-05-25 13:21:51.290437061 +0000 UTC m=+4146.948768466"
May 25 13:21:52.035122 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:52.035088    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bc7wq_6ebba530-a3c2-4deb-918c-b0f9ca60c101/serve-healthcheck-canary/0.log"
May 25 13:21:52.571927 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:52.571889    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5nx7_4818e986-b543-4e21-b438-69b2f3003855/kube-rbac-proxy/0.log"
May 25 13:21:52.592393 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:52.592363    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5nx7_4818e986-b543-4e21-b438-69b2f3003855/exporter/0.log"
May 25 13:21:52.612456 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:52.612417    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5nx7_4818e986-b543-4e21-b438-69b2f3003855/extractor/0.log"
May 25 13:21:54.634638 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:54.634611    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b94ff949c-ngfp8_8d7a3f82-6703-458d-a7eb-29f516883400/manager/0.log"
May 25 13:21:55.075472 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:55.075441    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-qv9q8_ba9fee2f-f1aa-4162-af7a-df874a53ba9b/manager/0.log"
May 25 13:21:57.287093 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:21:57.287062    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-72zf6"
May 25 13:22:00.351147 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.351112    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7hkxn_994d3f70-8d53-4232-9eec-f97966e0fa11/kube-multus/0.log"
May 25 13:22:00.742578 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.742552    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sfdq_e1367fb0-fb43-4995-9af7-9be8baf478af/kube-multus-additional-cni-plugins/0.log"
May 25 13:22:00.766802 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.766771    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sfdq_e1367fb0-fb43-4995-9af7-9be8baf478af/egress-router-binary-copy/0.log"
May 25 13:22:00.787502 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.787467    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sfdq_e1367fb0-fb43-4995-9af7-9be8baf478af/cni-plugins/0.log"
May 25 13:22:00.810380 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.810356    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sfdq_e1367fb0-fb43-4995-9af7-9be8baf478af/bond-cni-plugin/0.log"
May 25 13:22:00.838036 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.838006    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sfdq_e1367fb0-fb43-4995-9af7-9be8baf478af/routeoverride-cni/0.log"
May 25 13:22:00.859385 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.859357    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sfdq_e1367fb0-fb43-4995-9af7-9be8baf478af/whereabouts-cni-bincopy/0.log"
May 25 13:22:00.917976 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.917890    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sfdq_e1367fb0-fb43-4995-9af7-9be8baf478af/whereabouts-cni/0.log"
May 25 13:22:00.964109 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.964080    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-22jr7_037bfb47-48b6-4248-9537-b2da35bdf80f/network-metrics-daemon/0.log"
May 25 13:22:00.988580 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:00.988545    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-22jr7_037bfb47-48b6-4248-9537-b2da35bdf80f/kube-rbac-proxy/0.log"
May 25 13:22:02.883104 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:02.883072    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrrrl_96d029b0-6c0f-496a-b1f0-5a651825fc29/ovn-controller/0.log"
May 25 13:22:02.939022 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:02.938984    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrrrl_96d029b0-6c0f-496a-b1f0-5a651825fc29/ovn-acl-logging/0.log"
May 25 13:22:02.963779 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:02.963750    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrrrl_96d029b0-6c0f-496a-b1f0-5a651825fc29/kube-rbac-proxy-node/0.log"
May 25 13:22:02.996755 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:02.996724    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrrrl_96d029b0-6c0f-496a-b1f0-5a651825fc29/kube-rbac-proxy-ovn-metrics/0.log"
May 25 13:22:03.033731 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:03.033694    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrrrl_96d029b0-6c0f-496a-b1f0-5a651825fc29/northd/0.log"
May 25 13:22:03.114951 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:03.114911    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrrrl_96d029b0-6c0f-496a-b1f0-5a651825fc29/nbdb/0.log"
May 25 13:22:03.171159 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:03.171064    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrrrl_96d029b0-6c0f-496a-b1f0-5a651825fc29/sbdb/0.log"
May 25 13:22:03.361146 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:03.361118    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrrrl_96d029b0-6c0f-496a-b1f0-5a651825fc29/ovnkube-controller/0.log"
May 25 13:22:04.580094 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:04.580017    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-rvhv8_0a521353-c135-45c9-a6f1-0d36c801505d/network-check-target-container/0.log"
May 25 13:22:05.522702 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:05.522670    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2d962_782e7027-de68-4d9f-83de-04b38965cd87/iptables-alerter/0.log"
May 25 13:22:06.271967 ip-10-0-141-91 kubenswrapper[2562]: I0525 13:22:06.271929    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-tmtnm_249e00b6-ef97-49ed-8be8-069779eace7d/tuned/0.log"