May 20 15:31:51.047731 ip-10-0-140-154 systemd[1]: Starting Kubernetes Kubelet...
May 20 15:31:51.491090 ip-10-0-140-154 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:51.491090 ip-10-0-140-154 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 15:31:51.491090 ip-10-0-140-154 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:51.491090 ip-10-0-140-154 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 15:31:51.491090 ip-10-0-140-154 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:51.493526 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.493357    2570 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 15:31:51.496913 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496892    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:51.496913 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496912    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496916    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496920    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496924    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496928    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496932    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496935    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496939    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496942    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496945    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496948    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496950    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496953    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496957    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496959    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496962    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496964    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496967    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496970    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496972    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:51.496990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496975    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496978    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496980    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496985    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496988    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496991    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496994    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.496997    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497001    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497003    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497008    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497012    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497015    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497018    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497022    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497025    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497027    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497030    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497033    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:51.497532 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497035    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497038    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497041    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497044    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497046    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497050    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497052    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497057    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497061    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497064    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497067    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497069    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497072    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497074    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497077    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497081    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497084    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497086    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497089    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:51.497990 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497091    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497094    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497097    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497099    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497102    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497104    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497107    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497111    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497113    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497116    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497119    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497121    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497125    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497128    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497130    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497133    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497137    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497139    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497142    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497145    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:51.498471 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497147    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497150    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497154    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497157    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497160    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497162    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497165    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497613    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497619    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497623    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497625    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497628    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497631    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497634    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497638    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497642    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497646    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497649    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497652    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:51.498946 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497656    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497658    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497661    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497663    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497666    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497669    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497672    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497675    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497677    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497680    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497682    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497685    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497688    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497692    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497695    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497697    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497700    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497703    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497706    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:51.499425 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497708    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497711    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497714    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497717    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497719    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497722    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497724    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497727    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497730    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497733    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497735    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497738    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497741    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497743    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497746    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497749    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497751    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497754    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497757    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497759    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:51.499891 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497762    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497764    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497767    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497769    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497772    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497775    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497777    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497779    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497783    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497785    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497787    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497790    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497793    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497795    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497799    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497802    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497805    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497808    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497811    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497814    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:51.500380 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497816    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497819    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497821    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497823    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497826    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497828    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497831    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497834    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497836    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497839    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497842    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497845    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497847    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497850    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.497853    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498474    2570 flags.go:64] FLAG: --address="0.0.0.0"
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498486    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498494    2570 flags.go:64] FLAG: --anonymous-auth="true"
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498500    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498510    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498513    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 15:31:51.500861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498518    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498523    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498526    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498530    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498534    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498537    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498540    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498544    2570 flags.go:64] FLAG: --cgroup-root=""
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498547    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498550    2570 flags.go:64] FLAG: --client-ca-file=""
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498553    2570 flags.go:64] FLAG: --cloud-config=""
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498556    2570 flags.go:64] FLAG: --cloud-provider="external"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498559    2570 flags.go:64] FLAG: --cluster-dns="[]"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498564    2570 flags.go:64] FLAG: --cluster-domain=""
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498567    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498571    2570 flags.go:64] FLAG: --config-dir=""
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498573    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498577    2570 flags.go:64] FLAG: --container-log-max-files="5"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498582    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498585    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498589    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498592    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498596    2570 flags.go:64] FLAG: --contention-profiling="false"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498599    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 15:31:51.501402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498602    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498605    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498608    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498612    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498616    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498619    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498622    2570 flags.go:64] FLAG: --enable-load-reader="false"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498626    2570 flags.go:64] FLAG: --enable-server="true"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498628    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498635    2570 flags.go:64] FLAG: --event-burst="100"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498638    2570 flags.go:64] FLAG: --event-qps="50"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498641    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498644    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498648    2570 flags.go:64] FLAG: --eviction-hard=""
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498652    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498655    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498658    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498661    2570 flags.go:64] FLAG: --eviction-soft=""
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498664    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498667    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498671    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498674    2570 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498677    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498680    2570 flags.go:64] FLAG: --fail-swap-on="true"
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498683    2570 flags.go:64] FLAG: --feature-gates=""
May 20 15:31:51.501980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498687    2570 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498691    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498694    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498698    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498701    2570 flags.go:64] FLAG: --healthz-port="10248"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498704    2570 flags.go:64] FLAG: --help="false"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498707    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-140-154.ec2.internal"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498711    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498714    2570 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498717    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498721    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498724    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498727    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498730    2570 flags.go:64] FLAG: --image-service-endpoint=""
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498733    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498736    2570 flags.go:64] FLAG: --kube-api-burst="100"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498739    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498742    2570 flags.go:64] FLAG: --kube-api-qps="50"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498746    2570 flags.go:64] FLAG: --kube-reserved=""
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498749    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498752    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498755    2570 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498757    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498760    2570 flags.go:64] FLAG: --lock-file=""
May 20 15:31:51.502598 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498763    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498766    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498769    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498775    2570 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498778    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498781    2570 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498784    2570 flags.go:64] FLAG: --logging-format="text"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498787    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498790    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498793    2570 flags.go:64] FLAG: --manifest-url=""
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498796    2570 flags.go:64] FLAG: --manifest-url-header=""
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498801    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498804    2570 flags.go:64] FLAG: --max-open-files="1000000"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498809    2570 flags.go:64] FLAG: --max-pods="110"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498812    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498815    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498818    2570 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498821    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498824    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498827    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498830    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498839    2570 flags.go:64] FLAG: --node-status-max-images="50"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498842    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498845    2570 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 15:31:51.503216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498848    2570 flags.go:64] FLAG: --pod-cidr=""
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498855    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498862    2570 flags.go:64] FLAG: --pod-manifest-path=""
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498865    2570 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498868    2570 flags.go:64] FLAG: --pods-per-core="0"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498871    2570 flags.go:64] FLAG: --port="10250"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498874    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498877    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0445accf3f9b86d35"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498880    2570 flags.go:64] FLAG: --qos-reserved=""
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498884    2570 flags.go:64] FLAG: --read-only-port="10255"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498887    2570 flags.go:64] FLAG: --register-node="true"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498890    2570 flags.go:64] FLAG: --register-schedulable="true"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498894    2570 flags.go:64] FLAG: --register-with-taints=""
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498898    2570 flags.go:64] FLAG: --registry-burst="10"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498901    2570 flags.go:64] FLAG: --registry-qps="5"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498904    2570 flags.go:64] FLAG: --reserved-cpus=""
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498906    2570 flags.go:64] FLAG: --reserved-memory=""
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498910    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498913    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498916    2570 flags.go:64] FLAG: --rotate-certificates="false"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498920    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498923    2570 flags.go:64] FLAG: --runonce="false"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498927    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498930    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498933    2570 flags.go:64] FLAG: --seccomp-default="false"
May 20 15:31:51.503800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498936    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498939    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498942    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498945    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498949    2570 flags.go:64] FLAG: --storage-driver-password="root"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498952    2570 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498955    2570 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498959    2570 flags.go:64] FLAG: --storage-driver-user="root"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498962    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498965    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498968    2570 flags.go:64] FLAG: --system-cgroups=""
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498971    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498977    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498980    2570 flags.go:64] FLAG: --tls-cert-file=""
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498983    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498987    2570 flags.go:64] FLAG: --tls-min-version=""
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498990    2570 flags.go:64] FLAG: --tls-private-key-file=""
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498993    2570 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.498995    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.499001    2570 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.499004    2570 flags.go:64] FLAG: --v="2"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.499008    2570 flags.go:64] FLAG: --version="false"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.499013    2570 flags.go:64] FLAG: --vmodule=""
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.499018    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.499021    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 15:31:51.504438 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499135    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499140    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499143    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499146    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499149    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499152    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499155    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499157    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499160    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499164    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499167    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499170    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499172    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499175    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499179    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499182    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499186    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499189    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499192    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:51.505486 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499195    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499199    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499202    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499205    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499208    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499210    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499213    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499218    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499221    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499223    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499226    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499228    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499231    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499234    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499236    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499239    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499241    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499244    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499247    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499249    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:51.506274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499252    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499254    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499257    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499259    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499262    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499264    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499267    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499269    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499272    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499275    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499277    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499280    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499283    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499285    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499288    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499290    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499293    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499295    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499298    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499305    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:51.507249 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499308    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499310    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499313    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499315    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499318    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499321    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499324    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499327    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499330    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499332    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499335    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499338    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499341    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499343    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499346    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499348    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499351    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499353    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499356    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499358    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:51.508160 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499376    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499379    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499381    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499384    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499386    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499389    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.499391    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.499397    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.506946    2570 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.506978    2570 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507064    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507074    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507080    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507086    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507091    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507096    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:51.508819 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507101    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507106    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507110    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507114    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507118    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507123    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507127    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507132    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507136    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507140    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507145    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507149    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507164    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507169    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507173    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507178    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507182    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507186    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507191    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:51.509319 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507196    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507203    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507213    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507219    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507224    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507229    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507245    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507250    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507255    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507259    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507264    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507268    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507272    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507277    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507281    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507285    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507290    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507294    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507298    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:51.509959 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507303    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507307    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507312    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507316    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507320    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507325    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507330    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507334    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507339    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507343    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507347    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507352    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507357    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507381    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507386    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507390    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507395    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507398    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507402    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507409    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:51.510773 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507423    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507428    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507433    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507438    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507442    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507446    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507451    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507455    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507459    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507464    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507468    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507473    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507477    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507481    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507486    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507491    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507495    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507499    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507504    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507509    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:51.511341 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507513    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507519    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.507529    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507752    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507764    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507769    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507775    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507780    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507784    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507791    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507795    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507800    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507805    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507822    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507827    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:51.511867 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507832    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507836    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507841    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507846    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507850    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507854    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507858    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507863    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507867    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507871    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507876    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507880    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507885    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507889    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507894    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507898    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507902    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507907    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507912    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507916    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:51.512245 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507920    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507925    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507929    2570 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507934    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507939    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507943    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507947    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507951    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507958    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507963    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507967    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507981    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507985    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507993    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.507999    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508005    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508009    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508014    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508019    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:51.512783 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508024    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508028    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508032    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508036    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508041    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508045    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508050    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508055    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508062    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508067    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508071    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508075    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508079    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508083    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508087    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508091    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508095    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508100    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508104    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508109    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:51.513254 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508113    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508117    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508122    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508127    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508131    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508148    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508153    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508158    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508162    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508166    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508170    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508174    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508178    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508182    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:51.508186    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:51.514133 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.508195    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:51.514608 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.509093    2570 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 15:31:51.514608 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.514541    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 15:31:51.515895 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.515881    2570 server.go:1019] "Starting client certificate rotation"
May 20 15:31:51.516012 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.515989    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:31:51.516056 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.516043    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:31:51.539524 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.539490    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:31:51.546867 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.546836    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:31:51.561594 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.561570    2570 log.go:25] "Validated CRI v1 runtime API"
May 20 15:31:51.567119 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.567095    2570 log.go:25] "Validated CRI v1 image API"
May 20 15:31:51.568392 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.568350    2570 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 15:31:51.574259 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.574233    2570 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:31:51.575631 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.575603    2570 fs.go:135] Filesystem UUIDs: map[56d2fc0d-e9ba-4de5-86b2-2231bcc751fd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bb99c67f-db10-4a66-93b2-d3386010aa50:/dev/nvme0n1p3]
May 20 15:31:51.575708 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.575632    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 15:31:51.580500 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.580350    2570 manager.go:217] Machine: {Timestamp:2026-05-20 15:31:51.579770293 +0000 UTC m=+0.409866205 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103999 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e2a6bd83e05e804f1915e82cbb08d SystemUUID:ec2e2a6b-d83e-05e8-04f1-915e82cbb08d BootID:9d2b3577-9351-4f83-9d23-0aff67684cb0 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:af:fa:fe:cf:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:af:fa:fe:cf:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:6b:ec:43:fb:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 15:31:51.581219 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.581205    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 15:31:51.581324 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.581310    2570 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 15:31:51.584752 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.584713    2570 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 15:31:51.584925 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.584755    2570 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-154.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 15:31:51.584973 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.584939    2570 topology_manager.go:138] "Creating topology manager with none policy"
May 20 15:31:51.584973 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.584952    2570 container_manager_linux.go:306] "Creating device plugin manager"
May 20 15:31:51.584973 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.584964    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:31:51.586322 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.586301    2570 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:31:51.587439 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.587428    2570 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:31:51.587559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.587549    2570 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 15:31:51.589821 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.589809    2570 kubelet.go:491] "Attempting to sync node with API server"
May 20 15:31:51.589859 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.589832    2570 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 15:31:51.589859 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.589845    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 15:31:51.589944 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.589878    2570 kubelet.go:397] "Adding apiserver pod source"
May 20 15:31:51.589944 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.589889    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 15:31:51.591504 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.591483    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:31:51.591565 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.591523    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:31:51.595742 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.595713    2570 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 15:31:51.597951 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.597928    2570 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 15:31:51.601170 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601145    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 15:31:51.601170 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601172    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601178    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601184    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601194    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601200    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601206    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601212    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601219    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601228    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601240    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 15:31:51.601296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.601246    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 15:31:51.602172 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.602160    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 15:31:51.602172 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.602172    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 15:31:51.604519 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.604496    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 20 15:31:51.604605 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.604498    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-154.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 20 15:31:51.605792 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.605771    2570 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 15:31:51.605889 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.605813    2570 server.go:1295] "Started kubelet"
May 20 15:31:51.605943 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.605914    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 15:31:51.605974 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.605915    2570 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 15:31:51.606007 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.605989    2570 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 15:31:51.606898 ip-10-0-140-154 systemd[1]: Started Kubernetes Kubelet.
May 20 15:31:51.607092 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.606966    2570 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-154.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 20 15:31:51.607244 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.607230    2570 server.go:317] "Adding debug handlers to kubelet server"
May 20 15:31:51.607621 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.607602    2570 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 15:31:51.612146 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.612127    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 15:31:51.612597 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.612583    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 15:31:51.612743 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.612723    2570 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 15:31:51.612743 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.612741    2570 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 15:31:51.612851 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.612817    2570 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 15:31:51.612948 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.612924    2570 reconstruct.go:97] "Volume reconstruction finished"
May 20 15:31:51.612948 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.612938    2570 reconciler.go:26] "Reconciler: start to sync state"
May 20 15:31:51.613426 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.613407    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:51.615289 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.615264    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hd7m8"
May 20 15:31:51.615401 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.615353    2570 factory.go:55] Registering systemd factory
May 20 15:31:51.615401 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.615394    2570 factory.go:223] Registration of the systemd container factory successfully
May 20 15:31:51.615924 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.615903    2570 factory.go:153] Registering CRI-O factory
May 20 15:31:51.616020 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.615927    2570 factory.go:223] Registration of the crio container factory successfully
May 20 15:31:51.616020 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.615996    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 15:31:51.616020 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.616016    2570 factory.go:103] Registering Raw factory
May 20 15:31:51.616172 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.616031    2570 manager.go:1196] Started watching for new ooms in manager
May 20 15:31:51.616400 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.616375    2570 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 15:31:51.617290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.617273    2570 manager.go:319] Starting recovery of all containers
May 20 15:31:51.618083 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.618044    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-154.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 20 15:31:51.618882 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.618851    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 20 15:31:51.619166 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.618224    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-154.ec2.internal.18b14fd6c345d6a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-154.ec2.internal,UID:ip-10-0-140-154.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-154.ec2.internal,},FirstTimestamp:2026-05-20 15:31:51.605786274 +0000 UTC m=+0.435882185,LastTimestamp:2026-05-20 15:31:51.605786274 +0000 UTC m=+0.435882185,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-154.ec2.internal,}"
May 20 15:31:51.622580 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.622547    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hd7m8"
May 20 15:31:51.629529 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.629405    2570 manager.go:324] Recovery completed
May 20 15:31:51.633768 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.633750    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.638643 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.638624    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.638766 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.638659    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.638766 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.638673    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.639257 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.639243    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 15:31:51.639304 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.639258    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 15:31:51.639304 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.639277    2570 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:31:51.641068 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.640990    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-154.ec2.internal.18b14fd6c53b2de1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-154.ec2.internal,UID:ip-10-0-140-154.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-154.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-154.ec2.internal,},FirstTimestamp:2026-05-20 15:31:51.638642145 +0000 UTC m=+0.468738061,LastTimestamp:2026-05-20 15:31:51.638642145 +0000 UTC m=+0.468738061,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-154.ec2.internal,}"
May 20 15:31:51.642211 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.642199    2570 policy_none.go:49] "None policy: Start"
May 20 15:31:51.642270 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.642217    2570 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 15:31:51.642270 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.642228    2570 state_mem.go:35] "Initializing new in-memory state store"
May 20 15:31:51.686115 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.686095    2570 manager.go:341] "Starting Device Plugin manager"
May 20 15:31:51.696760 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.686150    2570 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 15:31:51.696760 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.686162    2570 server.go:85] "Starting device plugin registration server"
May 20 15:31:51.696760 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.686477    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 15:31:51.696760 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.686491    2570 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 15:31:51.696760 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.686582    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 15:31:51.696760 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.686676    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 15:31:51.696760 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.686685    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 15:31:51.696760 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.688032    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 15:31:51.696760 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.688067    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:51.753391 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.753345    2570 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 15:31:51.754672 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.754652    2570 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 15:31:51.754745 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.754680    2570 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 15:31:51.754745 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.754703    2570 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 15:31:51.754745 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.754711    2570 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 15:31:51.754886 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.754757    2570 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 15:31:51.757392 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.757355    2570 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:51.787085 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.787043    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.788574 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.788555    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.788713 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.788594    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.788713 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.788616    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.788713 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.788651    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-154.ec2.internal"
May 20 15:31:51.799653 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.799624    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-154.ec2.internal"
May 20 15:31:51.799653 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.799655    2570 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-154.ec2.internal\": node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:51.820457 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.820424    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:51.854851 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.854818    2570 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"]
May 20 15:31:51.855003 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.854915    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.856496 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.856477    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.856618 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.856511    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.856618 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.856525    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.858785 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.858766    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.858923 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.858906    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
May 20 15:31:51.858975 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.858938    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.860227 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.860193    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.860227 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.860220    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.860354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.860243    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.860354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.860253    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.860354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.860227    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.860354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.860312    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.862421 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.862403    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
May 20 15:31:51.862512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.862432    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.863319 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.863302    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.863442 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.863337    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.863442 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.863355    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.883299 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.883271    2570 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-154.ec2.internal\" not found" node="ip-10-0-140-154.ec2.internal"
May 20 15:31:51.887940 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.887917    2570 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-154.ec2.internal\" not found" node="ip-10-0-140-154.ec2.internal"
May 20 15:31:51.914264 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.914227    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f67fa41ea753d2bc9b49c9d5beea3265-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"f67fa41ea753d2bc9b49c9d5beea3265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
May 20 15:31:51.914264 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.914263    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2511ff5939997c60f4a09153c60ac7a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-154.ec2.internal\" (UID: \"f2511ff5939997c60f4a09153c60ac7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
May 20 15:31:51.914482 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:51.914280    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f67fa41ea753d2bc9b49c9d5beea3265-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"f67fa41ea753d2bc9b49c9d5beea3265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
May 20 15:31:51.921037 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:51.921004    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.014715 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.014640    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2511ff5939997c60f4a09153c60ac7a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-154.ec2.internal\" (UID: \"f2511ff5939997c60f4a09153c60ac7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
May 20 15:31:52.014715 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.014677    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f67fa41ea753d2bc9b49c9d5beea3265-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"f67fa41ea753d2bc9b49c9d5beea3265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
May 20 15:31:52.014715 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.014697    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f67fa41ea753d2bc9b49c9d5beea3265-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"f67fa41ea753d2bc9b49c9d5beea3265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
May 20 15:31:52.014879 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.014725    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f67fa41ea753d2bc9b49c9d5beea3265-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"f67fa41ea753d2bc9b49c9d5beea3265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
May 20 15:31:52.014879 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.014733    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2511ff5939997c60f4a09153c60ac7a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-154.ec2.internal\" (UID: \"f2511ff5939997c60f4a09153c60ac7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
May 20 15:31:52.014879 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.014753    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f67fa41ea753d2bc9b49c9d5beea3265-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"f67fa41ea753d2bc9b49c9d5beea3265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
May 20 15:31:52.021775 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.021744    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.121852 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.121819    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.186069 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.186027    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
May 20 15:31:52.189578 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.189560    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
May 20 15:31:52.222626 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.222588    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.323190 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.323089    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.423634 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.423595    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.514932 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.514900    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 15:31:52.515558 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.515083    2570 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 15:31:52.524205 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.524169    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.612755 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.612673    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 15:31:52.624552 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.624517    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.625625 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.625586    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 15:26:51 +0000 UTC" deadline="2027-10-31 22:36:30.775427535 +0000 UTC"
May 20 15:31:52.625694 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.625626    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12703h4m38.14980534s"
May 20 15:31:52.637173 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.637129    2570 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:31:52.675623 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:52.675585    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2511ff5939997c60f4a09153c60ac7a.slice/crio-cab1e1e5d6c0175572e0d7195f59e44287babf9d4260f4b1bbd734d64ee871d7 WatchSource:0}: Error finding container cab1e1e5d6c0175572e0d7195f59e44287babf9d4260f4b1bbd734d64ee871d7: Status 404 returned error can't find the container with id cab1e1e5d6c0175572e0d7195f59e44287babf9d4260f4b1bbd734d64ee871d7
May 20 15:31:52.682550 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.682530    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:31:52.687679 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.687645    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-79ljn"
May 20 15:31:52.690189 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:52.690164    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67fa41ea753d2bc9b49c9d5beea3265.slice/crio-733d2696233803aa480db5f82d6de92e2f9f7f789d0edd67efffde19407c41f8 WatchSource:0}: Error finding container 733d2696233803aa480db5f82d6de92e2f9f7f789d0edd67efffde19407c41f8: Status 404 returned error can't find the container with id 733d2696233803aa480db5f82d6de92e2f9f7f789d0edd67efffde19407c41f8
May 20 15:31:52.697208 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.697183    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-79ljn"
May 20 15:31:52.725380 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.725338    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.758543 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.758487    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal" event={"ID":"f67fa41ea753d2bc9b49c9d5beea3265","Type":"ContainerStarted","Data":"733d2696233803aa480db5f82d6de92e2f9f7f789d0edd67efffde19407c41f8"}
May 20 15:31:52.759456 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.759433    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal" event={"ID":"f2511ff5939997c60f4a09153c60ac7a","Type":"ContainerStarted","Data":"cab1e1e5d6c0175572e0d7195f59e44287babf9d4260f4b1bbd734d64ee871d7"}
May 20 15:31:52.826065 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.826031    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.926237 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:52.926157    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:52.927508 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:52.927486    2570 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:53.027127 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.027084    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:53.110949 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.110914    2570 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:53.127841 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.127802    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
May 20 15:31:53.160706 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.160666    2570 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:53.213059 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.212943    2570 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
May 20 15:31:53.225784 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.225710    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:31:53.226838 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.226707    2570 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
May 20 15:31:53.237802 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.237772    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:31:53.590799 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.590708    2570 apiserver.go:52] "Watching apiserver"
May 20 15:31:53.596744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.596712    2570 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 15:31:53.597152 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.597127    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm","openshift-image-registry/node-ca-994td","openshift-multus/multus-additional-cni-plugins-2b4js","openshift-multus/network-metrics-daemon-6fh2l","kube-system/konnectivity-agent-2gh5q","kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nf5mx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal","openshift-multus/multus-zrst8","openshift-network-diagnostics/network-check-target-nlb7f","openshift-network-operator/iptables-alerter-2bmz8","openshift-ovn-kubernetes/ovnkube-node-cw6k2"]
May 20 15:31:53.599746 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.599714    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.602048 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.602022    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.603715 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.603691    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 15:31:53.603845 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.603772    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v4vll\""
May 20 15:31:53.603845 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.603692    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 15:31:53.603845 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.603699    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 15:31:53.604255 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.604238    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:31:53.605863 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.605842    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 15:31:53.606590 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.606567    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.606864 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.606836    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 15:31:53.606952 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.606914    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 15:31:53.607089 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.607071    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 15:31:53.607175 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.606841    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 15:31:53.607582 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.607336    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7crlh\""
May 20 15:31:53.607582 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.607429    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9dzbf\""
May 20 15:31:53.608006 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.607766    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 15:31:53.608140 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.608119    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 15:31:53.608772 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.608754    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:31:53.608891 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.608836    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:31:53.609397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.609376    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 15:31:53.609975 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.609958    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 15:31:53.610092 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.610016    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mrg9t\""
May 20 15:31:53.611087 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.611067    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.613090 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.613071    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 15:31:53.613207 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.613129    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 15:31:53.613260 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.613216    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6hmhq\""
May 20 15:31:53.613601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.613323    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 15:31:53.614402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.614158    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.618500 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.618323    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 15:31:53.618500 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.618358    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 15:31:53.618500 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.618443    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 15:31:53.618812 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.618762    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 15:31:53.619840 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.619821    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 15:31:53.620153 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.620123    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 15:31:53.620253 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.620210    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jtlh9\""
May 20 15:31:53.620535 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.620513    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.620631 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.620599    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:53.620693 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.620666    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:31:53.623058 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.622846    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 15:31:53.623058 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.622844    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s79xg\""
May 20 15:31:53.623058 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.622913    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 15:31:53.623058 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623013    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zrst8"
May 20 15:31:53.623271 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623134    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 15:31:53.623557 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623533    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.623670 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623564    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-device-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.623670 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623580    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-etc-selinux\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.623670 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623594    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-etc-openvswitch\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.623670 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623618    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnkkv\" (UniqueName: \"kubernetes.io/projected/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-kube-api-access-lnkkv\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.623670 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623643    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:31:53.623670 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623665    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c-agent-certs\") pod \"konnectivity-agent-2gh5q\" (UID: \"1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c\") " pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623688    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-run-openvswitch\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623733    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623779    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623807    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r982m\" (UniqueName: \"kubernetes.io/projected/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-kube-api-access-r982m\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623833    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-sys\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623856    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-lib-modules\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623887    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-sysctl-d\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623911    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623929    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-sys-fs\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623943    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-var-lib-kubelet\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.623957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623962    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11ed634e-dd4e-4349-aca3-9374d531eaa8-host-slash\") pod \"iptables-alerter-2bmz8\" (UID: \"11ed634e-dd4e-4349-aca3-9374d531eaa8\") " pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.623985    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07b55da0-8f81-4c9f-b838-877bffe19092-etc-tuned\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-env-overrides\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624031    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-systemd-units\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624045    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-log-socket\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624064    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-ovnkube-config\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624087    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-ovn-node-metrics-cert\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624110    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrpzz\" (UniqueName: \"kubernetes.io/projected/fc257d99-db10-4fcd-bb95-7b306683c019-kube-api-access-hrpzz\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624130    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07b55da0-8f81-4c9f-b838-877bffe19092-tmp\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624145    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-kubelet\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624158    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-ovnkube-script-lib\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624176    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-run\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624212    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-slash\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624254    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-cni-netd\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624289    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-run-ovn\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624327    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-registration-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.624495 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624420    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-modprobe-d\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624450    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-sysconfig\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624483    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-kubernetes\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624506    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-systemd\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624529    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-host\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624551    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-var-lib-openvswitch\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624572    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-cni-bin\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624595    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-cnibin\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624618    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-os-release\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624642    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-socket-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624664    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c-konnectivity-ca\") pod \"konnectivity-agent-2gh5q\" (UID: \"1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c\") " pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624687    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-run-netns\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624720    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624749    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxrp\" (UniqueName: \"kubernetes.io/projected/07b55da0-8f81-4c9f-b838-877bffe19092-kube-api-access-nkxrp\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624889    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-node-log\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624925    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11ed634e-dd4e-4349-aca3-9374d531eaa8-iptables-alerter-script\") pod \"iptables-alerter-2bmz8\" (UID: \"11ed634e-dd4e-4349-aca3-9374d531eaa8\") " pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-sysctl-conf\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.625301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624972    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 15:31:53.625869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.624978    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-system-cni-dir\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.625869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.625001    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmxq\" (UniqueName: \"kubernetes.io/projected/11ed634e-dd4e-4349-aca3-9374d531eaa8-kube-api-access-npmxq\") pod \"iptables-alerter-2bmz8\" (UID: \"11ed634e-dd4e-4349-aca3-9374d531eaa8\") " pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.625869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.625030    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-run-systemd\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.625869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.625052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.625869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.625078    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.625869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.625271    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2ptcw\""
May 20 15:31:53.698551 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.698510    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:26:52 +0000 UTC" deadline="2027-12-29 01:08:29.801305988 +0000 UTC"
May 20 15:31:53.698551 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.698546    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14097h36m36.102763897s"
May 20 15:31:53.713782 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.713754    2570 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 15:31:53.725456 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725420    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-system-cni-dir\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.725456 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725456    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-var-lib-cni-multus\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.725678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725485    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-run\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.725678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725551    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-slash\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.725678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725603    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-run\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.725678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725647    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-slash\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.725678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725654    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-cni-netd\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.725906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725698    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-run-ovn\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.725906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725734    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-cni-netd\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.725906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725732    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-registration-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.725906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725787    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-run-ovn\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.725906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725798    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-modprobe-d\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.725906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725817    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-registration-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.725906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725849    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-sysconfig\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.725906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725889    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-kubernetes\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.725906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725895    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-modprobe-d\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725914    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-systemd\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725940    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-host\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725954    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-systemd\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725978    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-var-lib-openvswitch\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.725987    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-kubernetes\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726004    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-cni-bin\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726031    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-sysconfig\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726032    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-cnibin\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726073    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-cnibin\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726078    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-host\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726072    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-var-lib-openvswitch\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726078    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-os-release\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726112    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-cni-bin\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726139    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-os-release\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726154    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-socket-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726185    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c-konnectivity-ca\") pod \"konnectivity-agent-2gh5q\" (UID: \"1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c\") " pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:31:53.726290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726208    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-run-netns\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726236    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36d114bd-013f-4b9e-a9cc-b201bc41039b-cni-binary-copy\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726308    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-socket-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726265    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-var-lib-cni-bin\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726356    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726410    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-run-netns\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726473    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726521    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-run-multus-certs\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726661    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t422g\" (UniqueName: \"kubernetes.io/projected/1e841a1f-f858-4bb3-8884-7fc59fe8858b-kube-api-access-t422g\") pod \"node-ca-994td\" (UID: \"1e841a1f-f858-4bb3-8884-7fc59fe8858b\") " pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726723    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxrp\" (UniqueName: \"kubernetes.io/projected/07b55da0-8f81-4c9f-b838-877bffe19092-kube-api-access-nkxrp\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726849    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-node-log\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.726908    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c-konnectivity-ca\") pod \"konnectivity-agent-2gh5q\" (UID: \"1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c\") " pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727015    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-daemon-config\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78f8r\" (UniqueName: \"kubernetes.io/projected/36d114bd-013f-4b9e-a9cc-b201bc41039b-kube-api-access-78f8r\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727055    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.727075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727071    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-node-log\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727106    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11ed634e-dd4e-4349-aca3-9374d531eaa8-iptables-alerter-script\") pod \"iptables-alerter-2bmz8\" (UID: \"11ed634e-dd4e-4349-aca3-9374d531eaa8\") " pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727176    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-sysctl-conf\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727203    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-os-release\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727228    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-var-lib-kubelet\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727252    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-system-cni-dir\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727291    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npmxq\" (UniqueName: \"kubernetes.io/projected/11ed634e-dd4e-4349-aca3-9374d531eaa8-kube-api-access-npmxq\") pod \"iptables-alerter-2bmz8\" (UID: \"11ed634e-dd4e-4349-aca3-9374d531eaa8\") " pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727322    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-sysctl-conf\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727356    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-run-systemd\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727411    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-system-cni-dir\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727451    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727486    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-run-systemd\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727504    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727559    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727585    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-device-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727614    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-etc-selinux\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727638    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-etc-openvswitch\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.727763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727677    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkkv\" (UniqueName: \"kubernetes.io/projected/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-kube-api-access-lnkkv\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727702    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-device-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727706    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-conf-dir\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727736    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727803    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.727863    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11ed634e-dd4e-4349-aca3-9374d531eaa8-iptables-alerter-script\") pod \"iptables-alerter-2bmz8\" (UID: \"11ed634e-dd4e-4349-aca3-9374d531eaa8\") " pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728009    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c-agent-certs\") pod \"konnectivity-agent-2gh5q\" (UID: \"1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c\") " pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728074    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728262    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-etc-selinux\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728266    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-run-openvswitch\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728291    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-etc-openvswitch\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728378    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728405    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-run-openvswitch\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728429    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728487    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.728512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728489    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728526    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r982m\" (UniqueName: \"kubernetes.io/projected/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-kube-api-access-r982m\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728571    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-sys\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728596    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-lib-modules\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728629    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-cni-dir\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728677    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-run-netns\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728704    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-sysctl-d\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728729    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-sys\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728730    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-run-k8s-cni-cncf-io\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728532    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728766    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-etc-kubernetes\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728728    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728802    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e841a1f-f858-4bb3-8884-7fc59fe8858b-serviceca\") pod \"node-ca-994td\" (UID: \"1e841a1f-f858-4bb3-8884-7fc59fe8858b\") " pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728840    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728868    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-etc-sysctl-d\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728867    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-sys-fs\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728905    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-lib-modules\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.729178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728928    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-var-lib-kubelet\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.728966    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbzwq\" (UniqueName: \"kubernetes.io/projected/06532912-05d4-4680-bf74-819d7d4a36a0-kube-api-access-fbzwq\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729002    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729015    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07b55da0-8f81-4c9f-b838-877bffe19092-var-lib-kubelet\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729040    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc257d99-db10-4fcd-bb95-7b306683c019-sys-fs\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729063    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11ed634e-dd4e-4349-aca3-9374d531eaa8-host-slash\") pod \"iptables-alerter-2bmz8\" (UID: \"11ed634e-dd4e-4349-aca3-9374d531eaa8\") " pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729090    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07b55da0-8f81-4c9f-b838-877bffe19092-etc-tuned\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729126    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11ed634e-dd4e-4349-aca3-9374d531eaa8-host-slash\") pod \"iptables-alerter-2bmz8\" (UID: \"11ed634e-dd4e-4349-aca3-9374d531eaa8\") " pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729121    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-env-overrides\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729171    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-cnibin\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729194    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e841a1f-f858-4bb3-8884-7fc59fe8858b-host\") pod \"node-ca-994td\" (UID: \"1e841a1f-f858-4bb3-8884-7fc59fe8858b\") " pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729241    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-systemd-units\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729267    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-log-socket\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729311    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-ovnkube-config\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729329    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-systemd-units\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729348    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-log-socket\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729359    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-ovn-node-metrics-cert\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.729958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729443    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-hostroot\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.730771 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729492    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrpzz\" (UniqueName: \"kubernetes.io/projected/fc257d99-db10-4fcd-bb95-7b306683c019-kube-api-access-hrpzz\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.730771 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729521    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07b55da0-8f81-4c9f-b838-877bffe19092-tmp\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.730771 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729574    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-kubelet\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.730771 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729641    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-ovnkube-script-lib\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.730771 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729672    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-socket-dir-parent\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.730771 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729673    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-env-overrides\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.730771 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.729741    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-host-kubelet\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.731090 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.730883    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-ovnkube-config\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.731090 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.730910    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-ovnkube-script-lib\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.733948 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.733905    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07b55da0-8f81-4c9f-b838-877bffe19092-tmp\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.735039 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.734998    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:53.735039 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.735043    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:53.735226 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.735058    2570 projected.go:194] Error preparing data for projected volume kube-api-access-pnld9 for pod openshift-network-diagnostics/network-check-target-nlb7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:53.735226 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.735215    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9 podName:f946a720-3300-4953-a930-3f99f4b9443c nodeName:}" failed. No retries permitted until 2026-05-20 15:31:54.235156023 +0000 UTC m=+3.065251940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pnld9" (UniqueName: "kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9") pod "network-check-target-nlb7f" (UID: "f946a720-3300-4953-a930-3f99f4b9443c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:53.736455 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.736399    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c-agent-certs\") pod \"konnectivity-agent-2gh5q\" (UID: \"1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c\") " pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:31:53.736907 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.736859    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07b55da0-8f81-4c9f-b838-877bffe19092-etc-tuned\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.737499 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.737419    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-ovn-node-metrics-cert\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.738456 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.738429    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxrp\" (UniqueName: \"kubernetes.io/projected/07b55da0-8f81-4c9f-b838-877bffe19092-kube-api-access-nkxrp\") pod \"tuned-nf5mx\" (UID: \"07b55da0-8f81-4c9f-b838-877bffe19092\") " pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.738682 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.738654    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r982m\" (UniqueName: \"kubernetes.io/projected/01c8fe5f-0c58-4e0b-b213-d815ec0448bd-kube-api-access-r982m\") pod \"multus-additional-cni-plugins-2b4js\" (UID: \"01c8fe5f-0c58-4e0b-b213-d815ec0448bd\") " pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.738810 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.738790    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmxq\" (UniqueName: \"kubernetes.io/projected/11ed634e-dd4e-4349-aca3-9374d531eaa8-kube-api-access-npmxq\") pod \"iptables-alerter-2bmz8\" (UID: \"11ed634e-dd4e-4349-aca3-9374d531eaa8\") " pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.738882 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.738861    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnkkv\" (UniqueName: \"kubernetes.io/projected/cfeb4d5d-7e60-4d62-b083-cc887b4163ea-kube-api-access-lnkkv\") pod \"ovnkube-node-cw6k2\" (UID: \"cfeb4d5d-7e60-4d62-b083-cc887b4163ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.739570 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.739531    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrpzz\" (UniqueName: \"kubernetes.io/projected/fc257d99-db10-4fcd-bb95-7b306683c019-kube-api-access-hrpzz\") pod \"aws-ebs-csi-driver-node-bnfpm\" (UID: \"fc257d99-db10-4fcd-bb95-7b306683c019\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.831087 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831045    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-conf-dir\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831112    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-cni-dir\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831139    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-run-netns\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831160    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-run-k8s-cni-cncf-io\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831167    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-conf-dir\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831183    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-etc-kubernetes\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831225    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-run-k8s-cni-cncf-io\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831237    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-run-netns\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831244    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-cni-dir\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831226    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e841a1f-f858-4bb3-8884-7fc59fe8858b-serviceca\") pod \"node-ca-994td\" (UID: \"1e841a1f-f858-4bb3-8884-7fc59fe8858b\") " pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831279    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-etc-kubernetes\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831316    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzwq\" (UniqueName: \"kubernetes.io/projected/06532912-05d4-4680-bf74-819d7d4a36a0-kube-api-access-fbzwq\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831377    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-cnibin\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831403    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e841a1f-f858-4bb3-8884-7fc59fe8858b-host\") pod \"node-ca-994td\" (UID: \"1e841a1f-f858-4bb3-8884-7fc59fe8858b\") " pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831481    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e841a1f-f858-4bb3-8884-7fc59fe8858b-host\") pod \"node-ca-994td\" (UID: \"1e841a1f-f858-4bb3-8884-7fc59fe8858b\") " pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831510    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-cnibin\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831538    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-hostroot\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831563    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-socket-dir-parent\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831584    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-system-cni-dir\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831591    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-hostroot\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831620    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-var-lib-cni-multus\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831632    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-system-cni-dir\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831679    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-var-lib-cni-multus\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831683    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-socket-dir-parent\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.831714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831718    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e841a1f-f858-4bb3-8884-7fc59fe8858b-serviceca\") pod \"node-ca-994td\" (UID: \"1e841a1f-f858-4bb3-8884-7fc59fe8858b\") " pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831737    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36d114bd-013f-4b9e-a9cc-b201bc41039b-cni-binary-copy\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831772    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-var-lib-cni-bin\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831801    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831851    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-var-lib-cni-bin\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831888    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-run-multus-certs\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.831903    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831915    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t422g\" (UniqueName: \"kubernetes.io/projected/1e841a1f-f858-4bb3-8884-7fc59fe8858b-kube-api-access-t422g\") pod \"node-ca-994td\" (UID: \"1e841a1f-f858-4bb3-8884-7fc59fe8858b\") " pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831948    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-daemon-config\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.831982    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78f8r\" (UniqueName: \"kubernetes.io/projected/36d114bd-013f-4b9e-a9cc-b201bc41039b-kube-api-access-78f8r\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.832004    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-run-multus-certs\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.832014    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-os-release\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:53.832062    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs podName:06532912-05d4-4680-bf74-819d7d4a36a0 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:54.332042169 +0000 UTC m=+3.162138081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs") pod "network-metrics-daemon-6fh2l" (UID: "06532912-05d4-4680-bf74-819d7d4a36a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.832073    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-os-release\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.832082    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-var-lib-kubelet\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832340 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.832174    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36d114bd-013f-4b9e-a9cc-b201bc41039b-host-var-lib-kubelet\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832987 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.832386    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36d114bd-013f-4b9e-a9cc-b201bc41039b-cni-binary-copy\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.832987 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.832702    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36d114bd-013f-4b9e-a9cc-b201bc41039b-multus-daemon-config\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.839137 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.839090    2570 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:53.841399 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.841308    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78f8r\" (UniqueName: \"kubernetes.io/projected/36d114bd-013f-4b9e-a9cc-b201bc41039b-kube-api-access-78f8r\") pod \"multus-zrst8\" (UID: \"36d114bd-013f-4b9e-a9cc-b201bc41039b\") " pod="openshift-multus/multus-zrst8"
May 20 15:31:53.841644 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.841590    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzwq\" (UniqueName: \"kubernetes.io/projected/06532912-05d4-4680-bf74-819d7d4a36a0-kube-api-access-fbzwq\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:53.841750 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.841715    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t422g\" (UniqueName: \"kubernetes.io/projected/1e841a1f-f858-4bb3-8884-7fc59fe8858b-kube-api-access-t422g\") pod \"node-ca-994td\" (UID: \"1e841a1f-f858-4bb3-8884-7fc59fe8858b\") " pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.911883 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.911848    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm"
May 20 15:31:53.920770 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.920740    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2b4js"
May 20 15:31:53.929726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.929697    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:31:53.934634 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.934608    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nf5mx"
May 20 15:31:53.944349 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.944320    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bmz8"
May 20 15:31:53.951139 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.951113    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:31:53.956928 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.956896    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-994td"
May 20 15:31:53.962791 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:53.962758    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zrst8"
May 20 15:31:54.272798 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:54.272762    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ed634e_dd4e_4349_aca3_9374d531eaa8.slice/crio-68f9247d16e18f1d024acc2107ff97917ebc3206d817edfa041736e503363298 WatchSource:0}: Error finding container 68f9247d16e18f1d024acc2107ff97917ebc3206d817edfa041736e503363298: Status 404 returned error can't find the container with id 68f9247d16e18f1d024acc2107ff97917ebc3206d817edfa041736e503363298
May 20 15:31:54.274501 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:54.274421    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfeb4d5d_7e60_4d62_b083_cc887b4163ea.slice/crio-e94056bafa4137332da708e96c8be2f703dfe4f999b47aded391d80ede41d8d2 WatchSource:0}: Error finding container e94056bafa4137332da708e96c8be2f703dfe4f999b47aded391d80ede41d8d2: Status 404 returned error can't find the container with id e94056bafa4137332da708e96c8be2f703dfe4f999b47aded391d80ede41d8d2
May 20 15:31:54.277026 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:54.277001    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e841a1f_f858_4bb3_8884_7fc59fe8858b.slice/crio-71813071d06e4a083b164642f16bf396bd329766743904bcc715996a3941727c WatchSource:0}: Error finding container 71813071d06e4a083b164642f16bf396bd329766743904bcc715996a3941727c: Status 404 returned error can't find the container with id 71813071d06e4a083b164642f16bf396bd329766743904bcc715996a3941727c
May 20 15:31:54.277573 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:54.277552    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b55da0_8f81_4c9f_b838_877bffe19092.slice/crio-6fb464a92d176c466871ab7acc92cc094d63284d1fe0459753e1f37747c2199a WatchSource:0}: Error finding container 6fb464a92d176c466871ab7acc92cc094d63284d1fe0459753e1f37747c2199a: Status 404 returned error can't find the container with id 6fb464a92d176c466871ab7acc92cc094d63284d1fe0459753e1f37747c2199a
May 20 15:31:54.279504 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:54.279482    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d114bd_013f_4b9e_a9cc_b201bc41039b.slice/crio-6dc224434187598dd46af214df2cae5b5f9a0882c73c6bd0a289eff48c311f52 WatchSource:0}: Error finding container 6dc224434187598dd46af214df2cae5b5f9a0882c73c6bd0a289eff48c311f52: Status 404 returned error can't find the container with id 6dc224434187598dd46af214df2cae5b5f9a0882c73c6bd0a289eff48c311f52
May 20 15:31:54.280569 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:54.280538    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0cd5cb_80d8_46cd_bd4f_bd941cb1d78c.slice/crio-54a97d7f224a8d6a84fe52d2bac65e08cdbbb348ce728db7a7a71740f18ae086 WatchSource:0}: Error finding container 54a97d7f224a8d6a84fe52d2bac65e08cdbbb348ce728db7a7a71740f18ae086: Status 404 returned error can't find the container with id 54a97d7f224a8d6a84fe52d2bac65e08cdbbb348ce728db7a7a71740f18ae086
May 20 15:31:54.281338 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:54.281271    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc257d99_db10_4fcd_bb95_7b306683c019.slice/crio-4425758fe7c8cff0f0dad3f69e1bbebac93399fd56240d2ed18210e6943d8247 WatchSource:0}: Error finding container 4425758fe7c8cff0f0dad3f69e1bbebac93399fd56240d2ed18210e6943d8247: Status 404 returned error can't find the container with id 4425758fe7c8cff0f0dad3f69e1bbebac93399fd56240d2ed18210e6943d8247
May 20 15:31:54.282709 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:31:54.282674    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c8fe5f_0c58_4e0b_b213_d815ec0448bd.slice/crio-07fd0746adb8800f1d49c3c0965c30f095f7488a6147c461ea1e43699bec0df9 WatchSource:0}: Error finding container 07fd0746adb8800f1d49c3c0965c30f095f7488a6147c461ea1e43699bec0df9: Status 404 returned error can't find the container with id 07fd0746adb8800f1d49c3c0965c30f095f7488a6147c461ea1e43699bec0df9
May 20 15:31:54.335874 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.335637    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:31:54.336023 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.335936    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:54.336023 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:54.335832    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:54.336023 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:54.335975    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:54.336023 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:54.336005    2570 projected.go:194] Error preparing data for projected volume kube-api-access-pnld9 for pod openshift-network-diagnostics/network-check-target-nlb7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:54.336180 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:54.336041    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:54.336180 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:54.336073    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9 podName:f946a720-3300-4953-a930-3f99f4b9443c nodeName:}" failed. No retries permitted until 2026-05-20 15:31:55.336053772 +0000 UTC m=+4.166149693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnld9" (UniqueName: "kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9") pod "network-check-target-nlb7f" (UID: "f946a720-3300-4953-a930-3f99f4b9443c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:54.336180 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:54.336095    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs podName:06532912-05d4-4680-bf74-819d7d4a36a0 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:55.336084698 +0000 UTC m=+4.166180603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs") pod "network-metrics-daemon-6fh2l" (UID: "06532912-05d4-4680-bf74-819d7d4a36a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:54.699146 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.698996    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:26:52 +0000 UTC" deadline="2027-11-02 13:30:36.989308602 +0000 UTC"
May 20 15:31:54.699146 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.699052    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12741h58m42.290262012s"
May 20 15:31:54.755664 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.754916    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:54.755664 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:54.755073    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:31:54.755664 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.755530    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:31:54.755664 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:54.755613    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:31:54.767841 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.767652    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerStarted","Data":"07fd0746adb8800f1d49c3c0965c30f095f7488a6147c461ea1e43699bec0df9"}
May 20 15:31:54.771701 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.771602    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2gh5q" event={"ID":"1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c","Type":"ContainerStarted","Data":"54a97d7f224a8d6a84fe52d2bac65e08cdbbb348ce728db7a7a71740f18ae086"}
May 20 15:31:54.777143 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.777084    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zrst8" event={"ID":"36d114bd-013f-4b9e-a9cc-b201bc41039b","Type":"ContainerStarted","Data":"6dc224434187598dd46af214df2cae5b5f9a0882c73c6bd0a289eff48c311f52"}
May 20 15:31:54.798569 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.798529    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerStarted","Data":"e94056bafa4137332da708e96c8be2f703dfe4f999b47aded391d80ede41d8d2"}
May 20 15:31:54.813739 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.813643    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bmz8" event={"ID":"11ed634e-dd4e-4349-aca3-9374d531eaa8","Type":"ContainerStarted","Data":"68f9247d16e18f1d024acc2107ff97917ebc3206d817edfa041736e503363298"}
May 20 15:31:54.842047 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.842008    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal" event={"ID":"f2511ff5939997c60f4a09153c60ac7a","Type":"ContainerStarted","Data":"4909b751a12f5dcc563521e49ef6ef73aa85678a0f357cff62a466ee5f846fe5"}
May 20 15:31:54.853986 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.853907    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm" event={"ID":"fc257d99-db10-4fcd-bb95-7b306683c019","Type":"ContainerStarted","Data":"4425758fe7c8cff0f0dad3f69e1bbebac93399fd56240d2ed18210e6943d8247"}
May 20 15:31:54.857854 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.857232    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal" podStartSLOduration=1.8572161390000002 podStartE2EDuration="1.857216139s" podCreationTimestamp="2026-05-20 15:31:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:31:54.856541785 +0000 UTC m=+3.686637709" watchObservedRunningTime="2026-05-20 15:31:54.857216139 +0000 UTC m=+3.687312062"
May 20 15:31:54.857854 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.857794    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nf5mx" event={"ID":"07b55da0-8f81-4c9f-b838-877bffe19092","Type":"ContainerStarted","Data":"6fb464a92d176c466871ab7acc92cc094d63284d1fe0459753e1f37747c2199a"}
May 20 15:31:54.867984 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.867946    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-994td" event={"ID":"1e841a1f-f858-4bb3-8884-7fc59fe8858b","Type":"ContainerStarted","Data":"71813071d06e4a083b164642f16bf396bd329766743904bcc715996a3941727c"}
May 20 15:31:54.883177 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.882346    2570 generic.go:358] "Generic (PLEG): container finished" podID="f67fa41ea753d2bc9b49c9d5beea3265" containerID="e5fd11a3dac1f8e7a1b753623a47fa790bef603cae5f4cb0fa9d851b754a8a58" exitCode=0
May 20 15:31:54.883177 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:54.882412    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal" event={"ID":"f67fa41ea753d2bc9b49c9d5beea3265","Type":"ContainerDied","Data":"e5fd11a3dac1f8e7a1b753623a47fa790bef603cae5f4cb0fa9d851b754a8a58"}
May 20 15:31:55.348053 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:55.347963    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:55.348053 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:55.348034    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:31:55.348276 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:55.348182    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:55.348276 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:55.348202    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:55.348276 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:55.348216    2570 projected.go:194] Error preparing data for projected volume kube-api-access-pnld9 for pod openshift-network-diagnostics/network-check-target-nlb7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:55.348454 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:55.348278    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9 podName:f946a720-3300-4953-a930-3f99f4b9443c nodeName:}" failed. No retries permitted until 2026-05-20 15:31:57.34825836 +0000 UTC m=+6.178354271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnld9" (UniqueName: "kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9") pod "network-check-target-nlb7f" (UID: "f946a720-3300-4953-a930-3f99f4b9443c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:55.348719 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:55.348701    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:55.348824 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:55.348757    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs podName:06532912-05d4-4680-bf74-819d7d4a36a0 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:57.348740654 +0000 UTC m=+6.178836558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs") pod "network-metrics-daemon-6fh2l" (UID: "06532912-05d4-4680-bf74-819d7d4a36a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:55.888902 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:55.888845    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal" event={"ID":"f67fa41ea753d2bc9b49c9d5beea3265","Type":"ContainerStarted","Data":"f9bc18b4f0f9ab4201bfbbc22806bbf51865e0f77591174d7731fac1ce594223"}
May 20 15:31:55.905744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:55.905685    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal" podStartSLOduration=2.9056666140000003 podStartE2EDuration="2.905666614s" podCreationTimestamp="2026-05-20 15:31:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:31:55.904996711 +0000 UTC m=+4.735092633" watchObservedRunningTime="2026-05-20 15:31:55.905666614 +0000 UTC m=+4.735762536"
May 20 15:31:56.755678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:56.755101    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:56.755678 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:56.755245    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:31:56.755678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:56.755544    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:31:56.755678 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:56.755647    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:31:57.366778 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:57.366735    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:31:57.367271 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:57.366832    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:57.367271 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:57.366968    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:57.367271 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:57.367045    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs podName:06532912-05d4-4680-bf74-819d7d4a36a0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:01.367016135 +0000 UTC m=+10.197112036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs") pod "network-metrics-daemon-6fh2l" (UID: "06532912-05d4-4680-bf74-819d7d4a36a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:57.367646 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:57.367516    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:57.367646 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:57.367540    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:57.367646 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:57.367554    2570 projected.go:194] Error preparing data for projected volume kube-api-access-pnld9 for pod openshift-network-diagnostics/network-check-target-nlb7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:57.367646 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:57.367608    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9 podName:f946a720-3300-4953-a930-3f99f4b9443c nodeName:}" failed. No retries permitted until 2026-05-20 15:32:01.367591937 +0000 UTC m=+10.197687840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnld9" (UniqueName: "kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9") pod "network-check-target-nlb7f" (UID: "f946a720-3300-4953-a930-3f99f4b9443c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:58.755612 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:58.755571    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:31:58.756083 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:58.755695    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:31:58.756083 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:31:58.755878    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:31:58.756083 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:31:58.755982    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:00.755991 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:00.755941    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:00.755991 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:00.755961    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:00.756544 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:00.756080    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:00.756544 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:00.756222    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:01.403266 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:01.402745    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:01.403266 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:01.402831    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:01.403266 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:01.402947    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:01.403266 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:01.402959    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:01.403266 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:01.402987    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:01.403266 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:01.403003    2570 projected.go:194] Error preparing data for projected volume kube-api-access-pnld9 for pod openshift-network-diagnostics/network-check-target-nlb7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:01.403266 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:01.403029    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs podName:06532912-05d4-4680-bf74-819d7d4a36a0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:09.4030007 +0000 UTC m=+18.233096604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs") pod "network-metrics-daemon-6fh2l" (UID: "06532912-05d4-4680-bf74-819d7d4a36a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:01.403266 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:01.403069    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9 podName:f946a720-3300-4953-a930-3f99f4b9443c nodeName:}" failed. No retries permitted until 2026-05-20 15:32:09.403052089 +0000 UTC m=+18.233148006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnld9" (UniqueName: "kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9") pod "network-check-target-nlb7f" (UID: "f946a720-3300-4953-a930-3f99f4b9443c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:02.754992 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:02.754960    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:02.755438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:02.754959    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:02.755438 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:02.755091    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:02.755438 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:02.755208    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:04.755164 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:04.755121    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:04.755629 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:04.755121    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:04.755629 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:04.755254    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:04.755629 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:04.755380    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:05.678497 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.678464    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-2mr9z"]
May 20 15:32:05.682641 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.682608    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.695261 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.695229    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 15:32:05.695444 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.695229    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 15:32:05.695444 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.695243    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h2q86\""
May 20 15:32:05.733823 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.733782    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf845\" (UniqueName: \"kubernetes.io/projected/ed665954-8c00-4145-b7de-3605ca754aac-kube-api-access-vf845\") pod \"node-resolver-2mr9z\" (UID: \"ed665954-8c00-4145-b7de-3605ca754aac\") " pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.734002 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.733855    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed665954-8c00-4145-b7de-3605ca754aac-hosts-file\") pod \"node-resolver-2mr9z\" (UID: \"ed665954-8c00-4145-b7de-3605ca754aac\") " pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.734002 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.733891    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed665954-8c00-4145-b7de-3605ca754aac-tmp-dir\") pod \"node-resolver-2mr9z\" (UID: \"ed665954-8c00-4145-b7de-3605ca754aac\") " pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.834421 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.834387    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed665954-8c00-4145-b7de-3605ca754aac-hosts-file\") pod \"node-resolver-2mr9z\" (UID: \"ed665954-8c00-4145-b7de-3605ca754aac\") " pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.834880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.834430    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed665954-8c00-4145-b7de-3605ca754aac-tmp-dir\") pod \"node-resolver-2mr9z\" (UID: \"ed665954-8c00-4145-b7de-3605ca754aac\") " pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.834880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.834481    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vf845\" (UniqueName: \"kubernetes.io/projected/ed665954-8c00-4145-b7de-3605ca754aac-kube-api-access-vf845\") pod \"node-resolver-2mr9z\" (UID: \"ed665954-8c00-4145-b7de-3605ca754aac\") " pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.834880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.834537    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed665954-8c00-4145-b7de-3605ca754aac-hosts-file\") pod \"node-resolver-2mr9z\" (UID: \"ed665954-8c00-4145-b7de-3605ca754aac\") " pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.834880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.834863    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed665954-8c00-4145-b7de-3605ca754aac-tmp-dir\") pod \"node-resolver-2mr9z\" (UID: \"ed665954-8c00-4145-b7de-3605ca754aac\") " pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.849647 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.849607    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf845\" (UniqueName: \"kubernetes.io/projected/ed665954-8c00-4145-b7de-3605ca754aac-kube-api-access-vf845\") pod \"node-resolver-2mr9z\" (UID: \"ed665954-8c00-4145-b7de-3605ca754aac\") " pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:05.992045 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:05.991962    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2mr9z"
May 20 15:32:06.755843 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:06.755806    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:06.755843 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:06.755844    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:06.756162 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:06.755942    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:06.756162 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:06.756077    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:08.755660 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:08.755624    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:08.756185 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:08.755624    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:08.756185 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:08.755771    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:08.756185 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:08.755874    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:09.460516 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:09.460462    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:09.460684 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:09.460534    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:09.460684 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:09.460641    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:09.460684 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:09.460655    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:09.460684 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:09.460670    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:09.460684 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:09.460683    2570 projected.go:194] Error preparing data for projected volume kube-api-access-pnld9 for pod openshift-network-diagnostics/network-check-target-nlb7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:09.460872 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:09.460718    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs podName:06532912-05d4-4680-bf74-819d7d4a36a0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:25.460696949 +0000 UTC m=+34.290792861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs") pod "network-metrics-daemon-6fh2l" (UID: "06532912-05d4-4680-bf74-819d7d4a36a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:09.460872 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:09.460735    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9 podName:f946a720-3300-4953-a930-3f99f4b9443c nodeName:}" failed. No retries permitted until 2026-05-20 15:32:25.460728485 +0000 UTC m=+34.290824384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnld9" (UniqueName: "kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9") pod "network-check-target-nlb7f" (UID: "f946a720-3300-4953-a930-3f99f4b9443c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:10.755648 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:10.755609    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:10.756085 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:10.755611    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:10.756085 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:10.755728    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:10.756085 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:10.755815    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:11.896151 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:11.896102    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded665954_8c00_4145_b7de_3605ca754aac.slice/crio-dba471cfd1e479f8e54a684177c92a87573135863344ea8937fedfb7c604fb1e WatchSource:0}: Error finding container dba471cfd1e479f8e54a684177c92a87573135863344ea8937fedfb7c604fb1e: Status 404 returned error can't find the container with id dba471cfd1e479f8e54a684177c92a87573135863344ea8937fedfb7c604fb1e
May 20 15:32:11.924478 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:11.924442    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2mr9z" event={"ID":"ed665954-8c00-4145-b7de-3605ca754aac","Type":"ContainerStarted","Data":"dba471cfd1e479f8e54a684177c92a87573135863344ea8937fedfb7c604fb1e"}
May 20 15:32:12.755554 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.755187    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:12.755728 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.755214    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:12.755728 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:12.755602    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:12.755728 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:12.755715    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:12.927214 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.927175    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2mr9z" event={"ID":"ed665954-8c00-4145-b7de-3605ca754aac","Type":"ContainerStarted","Data":"a40f26b8addc18a385f415ef9ccc372a7e498a8b099d752f7e576e37b2b09dc5"}
May 20 15:32:12.928526 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.928506    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm" event={"ID":"fc257d99-db10-4fcd-bb95-7b306683c019","Type":"ContainerStarted","Data":"e377b20dd5e3d5e1a3f103e7eef11074a8b04dd2a31bc23ad98ac4bf482160e2"}
May 20 15:32:12.929662 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.929642    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nf5mx" event={"ID":"07b55da0-8f81-4c9f-b838-877bffe19092","Type":"ContainerStarted","Data":"8c2f9ba97380ac67290987d876e68d9eb9b6d4160b0f35ca4aabc37460fde938"}
May 20 15:32:12.930758 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.930737    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-994td" event={"ID":"1e841a1f-f858-4bb3-8884-7fc59fe8858b","Type":"ContainerStarted","Data":"95face6f3d006396405f0427fc3d0217f59797f5fd36729fdb27c19ee7d7e314"}
May 20 15:32:12.931961 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.931942    2570 generic.go:358] "Generic (PLEG): container finished" podID="01c8fe5f-0c58-4e0b-b213-d815ec0448bd" containerID="5339b9eba1dedc35119fb6efc73740bb8a71e091c4e29ff6429966378dc7950d" exitCode=0
May 20 15:32:12.932033 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.931970    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerDied","Data":"5339b9eba1dedc35119fb6efc73740bb8a71e091c4e29ff6429966378dc7950d"}
May 20 15:32:12.933234 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.933214    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2gh5q" event={"ID":"1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c","Type":"ContainerStarted","Data":"054480e04c89e93d3bd76162cec7f819f0acfe7e163f2bba0a09e3414edb5f46"}
May 20 15:32:12.934531 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.934511    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zrst8" event={"ID":"36d114bd-013f-4b9e-a9cc-b201bc41039b","Type":"ContainerStarted","Data":"836b0486a85ca55c505f04b618ad98e4263bd1e004cc7072bb9a1a6a85729a91"}
May 20 15:32:12.936810 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.936794    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:32:12.937087 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.937069    2570 generic.go:358] "Generic (PLEG): container finished" podID="cfeb4d5d-7e60-4d62-b083-cc887b4163ea" containerID="c7083b01df2b102621865c468d3dd3f3c55bf6c71b44cdaa8f2eaa7596774d30" exitCode=1
May 20 15:32:12.937134 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.937100    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerStarted","Data":"a12ef7db8c667b49a1231155762808bad047b2fb856f6d8c3a80791a366af2f0"}
May 20 15:32:12.937134 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.937116    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerStarted","Data":"72a9661d7ed30dfe2dcd6ddf6c68006d820132995aba71a3c86fade48367d4ac"}
May 20 15:32:12.937134 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.937125    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerStarted","Data":"5b5b4933cb37bcfcb436c81c8c31bb212eda96e0913f1dfb314973cad985c3ea"}
May 20 15:32:12.937134 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.937134    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerStarted","Data":"c16638e7d42f32edbbb5faef7d1929d4439c0dba48cfc3330bfa3e16e60fc3c1"}
May 20 15:32:12.937258 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.937145    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerDied","Data":"c7083b01df2b102621865c468d3dd3f3c55bf6c71b44cdaa8f2eaa7596774d30"}
May 20 15:32:12.937258 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.937159    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerStarted","Data":"4f998e26619180adb0002150e25eb1b194793e339a647bd7b8b1961e0628bd04"}
May 20 15:32:12.943524 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.943491    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-2mr9z" podStartSLOduration=7.943480856 podStartE2EDuration="7.943480856s" podCreationTimestamp="2026-05-20 15:32:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:12.942844327 +0000 UTC m=+21.772940248" watchObservedRunningTime="2026-05-20 15:32:12.943480856 +0000 UTC m=+21.773576777"
May 20 15:32:12.960507 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.960467    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2gh5q" podStartSLOduration=8.169405791 podStartE2EDuration="21.960454443s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:54.283179951 +0000 UTC m=+3.113275851" lastFinishedPulling="2026-05-20 15:32:08.074228603 +0000 UTC m=+16.904324503" observedRunningTime="2026-05-20 15:32:12.959681179 +0000 UTC m=+21.789777100" watchObservedRunningTime="2026-05-20 15:32:12.960454443 +0000 UTC m=+21.790550363"
May 20 15:32:12.984151 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:12.984112    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nf5mx" podStartSLOduration=4.372049006 podStartE2EDuration="21.984101569s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:54.279407661 +0000 UTC m=+3.109503559" lastFinishedPulling="2026-05-20 15:32:11.891460209 +0000 UTC m=+20.721556122" observedRunningTime="2026-05-20 15:32:12.982293584 +0000 UTC m=+21.812389504" watchObservedRunningTime="2026-05-20 15:32:12.984101569 +0000 UTC m=+21.814197542"
May 20 15:32:13.033592 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.033548    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-994td" podStartSLOduration=3.422572211 podStartE2EDuration="21.033535175s" podCreationTimestamp="2026-05-20 15:31:52 +0000 UTC" firstStartedPulling="2026-05-20 15:31:54.278609589 +0000 UTC m=+3.108705488" lastFinishedPulling="2026-05-20 15:32:11.889572538 +0000 UTC m=+20.719668452" observedRunningTime="2026-05-20 15:32:13.033308505 +0000 UTC m=+21.863404426" watchObservedRunningTime="2026-05-20 15:32:13.033535175 +0000 UTC m=+21.863631095"
May 20 15:32:13.052267 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.052224    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-zrst8" podStartSLOduration=3.388573483 podStartE2EDuration="21.052211117s" podCreationTimestamp="2026-05-20 15:31:52 +0000 UTC" firstStartedPulling="2026-05-20 15:31:54.281469783 +0000 UTC m=+3.111565697" lastFinishedPulling="2026-05-20 15:32:11.945107417 +0000 UTC m=+20.775203331" observedRunningTime="2026-05-20 15:32:13.051603064 +0000 UTC m=+21.881698986" watchObservedRunningTime="2026-05-20 15:32:13.052211117 +0000 UTC m=+21.882307057"
May 20 15:32:13.239783 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.239756    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 15:32:13.285407 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.285324    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2mr9z_ed665954-8c00-4145-b7de-3605ca754aac/dns-node-resolver/0.log"
May 20 15:32:13.698772 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.698594    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T15:32:13.239779532Z","UUID":"e476e037-f093-4a1f-9273-74f1d17fe0ef","Handler":null,"Name":"","Endpoint":""}
May 20 15:32:13.700780 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.700666    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 15:32:13.700780 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.700703    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 15:32:13.943073 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.942320    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bmz8" event={"ID":"11ed634e-dd4e-4349-aca3-9374d531eaa8","Type":"ContainerStarted","Data":"11cbeddaa9b6d24b2c5bd4d41153d8e7485b0ef92c0b6ba736a067b932c23117"}
May 20 15:32:13.945476 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.945414    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm" event={"ID":"fc257d99-db10-4fcd-bb95-7b306683c019","Type":"ContainerStarted","Data":"30274737bfd6ab2fa6d6a4837b38cf6257b5472ec15e594affc9129e4991f423"}
May 20 15:32:13.960908 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:13.960797    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2bmz8" podStartSLOduration=5.3475845 podStartE2EDuration="22.960781504s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:54.275918238 +0000 UTC m=+3.106014139" lastFinishedPulling="2026-05-20 15:32:11.88911524 +0000 UTC m=+20.719211143" observedRunningTime="2026-05-20 15:32:13.960230848 +0000 UTC m=+22.790326771" watchObservedRunningTime="2026-05-20 15:32:13.960781504 +0000 UTC m=+22.790877452"
May 20 15:32:14.452587 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:14.452330    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-994td_1e841a1f-f858-4bb3-8884-7fc59fe8858b/node-ca/0.log"
May 20 15:32:14.755376 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:14.755334    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:14.755702 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:14.755482    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:14.755702 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:14.755509    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:14.755702 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:14.755595    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:14.950510 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:14.950485    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:32:14.950934 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:14.950906    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerStarted","Data":"9e9f44ae99da40793e5b66a3fb6d49224502fcc2a5fdd936795cae76882f6621"}
May 20 15:32:14.953266 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:14.953234    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm" event={"ID":"fc257d99-db10-4fcd-bb95-7b306683c019","Type":"ContainerStarted","Data":"2df8a01e16b12359f15d944c0d277e5744607069b5097fdc49bdf9a23a3e85c8"}
May 20 15:32:14.973014 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:14.972951    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnfpm" podStartSLOduration=4.021496112 podStartE2EDuration="23.972929186s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:54.284900418 +0000 UTC m=+3.114996323" lastFinishedPulling="2026-05-20 15:32:14.236333492 +0000 UTC m=+23.066429397" observedRunningTime="2026-05-20 15:32:14.972415544 +0000 UTC m=+23.802511466" watchObservedRunningTime="2026-05-20 15:32:14.972929186 +0000 UTC m=+23.803025109"
May 20 15:32:15.170532 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:15.170498    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:32:15.171187 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:15.171163    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:32:15.955876 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:15.955840    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:32:15.956539 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:15.956275    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2gh5q"
May 20 15:32:16.755569 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:16.755528    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:16.755569 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:16.755561    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:16.755776 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:16.755647    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:16.755776 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:16.755746    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:16.959051 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:16.958790    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerStarted","Data":"e5df66802f549c8ea3f90d5ae07f3565a847044d0f462f490c9b3073452ed42b"}
May 20 15:32:16.963335 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:16.962960    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:32:16.963927 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:16.963443    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerStarted","Data":"227d51f8dc16a5633cb5bea16ee539ad431fb6ec1dbe950356e80c01aa06d523"}
May 20 15:32:16.963927 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:16.963864    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:32:16.963927 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:16.963889    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:32:16.964328 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:16.963940    2570 scope.go:117] "RemoveContainer" containerID="c7083b01df2b102621865c468d3dd3f3c55bf6c71b44cdaa8f2eaa7596774d30"
May 20 15:32:16.982311 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:16.982189    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:32:17.968870 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:17.968834    2570 generic.go:358] "Generic (PLEG): container finished" podID="01c8fe5f-0c58-4e0b-b213-d815ec0448bd" containerID="e5df66802f549c8ea3f90d5ae07f3565a847044d0f462f490c9b3073452ed42b" exitCode=0
May 20 15:32:17.969341 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:17.968920    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerDied","Data":"e5df66802f549c8ea3f90d5ae07f3565a847044d0f462f490c9b3073452ed42b"}
May 20 15:32:17.975350 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:17.975330    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:32:17.975666 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:17.975641    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" event={"ID":"cfeb4d5d-7e60-4d62-b083-cc887b4163ea","Type":"ContainerStarted","Data":"194068b974df01e358d846a57477e75faf61feb671acc29135dfd89b3be02a38"}
May 20 15:32:17.976195 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:17.976171    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:32:17.992565 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:17.992531    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:32:18.043657 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:18.043601    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2" podStartSLOduration=9.381209815 podStartE2EDuration="27.043584839s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:54.276453424 +0000 UTC m=+3.106549322" lastFinishedPulling="2026-05-20 15:32:11.938828439 +0000 UTC m=+20.768924346" observedRunningTime="2026-05-20 15:32:18.041751569 +0000 UTC m=+26.871847490" watchObservedRunningTime="2026-05-20 15:32:18.043584839 +0000 UTC m=+26.873680759"
May 20 15:32:18.755593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:18.755557    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:18.755720 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:18.755554    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:18.755773 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:18.755708    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:18.755815 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:18.755763    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:18.979245 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:18.979083    2570 generic.go:358] "Generic (PLEG): container finished" podID="01c8fe5f-0c58-4e0b-b213-d815ec0448bd" containerID="7f98f6a53a9e2381639a8baeda9c1a5dc5e41d4142c2511e456414213a0c48ac" exitCode=0
May 20 15:32:18.979667 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:18.979165    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerDied","Data":"7f98f6a53a9e2381639a8baeda9c1a5dc5e41d4142c2511e456414213a0c48ac"}
May 20 15:32:18.996032 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:18.996011    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6fh2l"]
May 20 15:32:18.996121 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:18.996095    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:18.996190 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:18.996175    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:19.008477 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:19.008415    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nlb7f"]
May 20 15:32:19.008576 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:19.008515    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:19.008641 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:19.008618    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:19.982482 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:19.982450    2570 generic.go:358] "Generic (PLEG): container finished" podID="01c8fe5f-0c58-4e0b-b213-d815ec0448bd" containerID="972a5d0e47a58ed8e31dc8fd1035b49ebe88e936b3bbefdee8f4133bdeb5d4db" exitCode=0
May 20 15:32:19.982824 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:19.982529    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerDied","Data":"972a5d0e47a58ed8e31dc8fd1035b49ebe88e936b3bbefdee8f4133bdeb5d4db"}
May 20 15:32:20.754988 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:20.754953    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:20.755191 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:20.754953    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:20.755191 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:20.755114    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:20.755191 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:20.755159    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:22.754956 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:22.754916    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:22.754956 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:22.754947    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:22.755551 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:22.755065    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:22.755551 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:22.755220    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:24.755055 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:24.755016    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:24.755733 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:24.755073    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:24.755733 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:24.755191    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:24.755733 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:24.755331    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:25.484147 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:25.484106    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:25.484331 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:25.484165    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:25.484331 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:25.484267    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:25.484331 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:25.484278    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:25.484331 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:25.484292    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:25.484331 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:25.484302    2570 projected.go:194] Error preparing data for projected volume kube-api-access-pnld9 for pod openshift-network-diagnostics/network-check-target-nlb7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:25.484535 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:25.484340    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs podName:06532912-05d4-4680-bf74-819d7d4a36a0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:57.484321035 +0000 UTC m=+66.314416939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs") pod "network-metrics-daemon-6fh2l" (UID: "06532912-05d4-4680-bf74-819d7d4a36a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:25.484535 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:25.484356    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9 podName:f946a720-3300-4953-a930-3f99f4b9443c nodeName:}" failed. No retries permitted until 2026-05-20 15:32:57.484350104 +0000 UTC m=+66.314446002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnld9" (UniqueName: "kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9") pod "network-check-target-nlb7f" (UID: "f946a720-3300-4953-a930-3f99f4b9443c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:25.998047 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:25.998007    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerStarted","Data":"ceb103dc28e383e514cd25f1799eac023d62c772549507867d5237cc7ca83237"}
May 20 15:32:26.755947 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:26.755911    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:26.756117 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:26.755911    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:26.756117 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:26.756027    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:26.756189 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:26.756147    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:27.001707 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:27.001671    2570 generic.go:358] "Generic (PLEG): container finished" podID="01c8fe5f-0c58-4e0b-b213-d815ec0448bd" containerID="ceb103dc28e383e514cd25f1799eac023d62c772549507867d5237cc7ca83237" exitCode=0
May 20 15:32:27.002154 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:27.001735    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerDied","Data":"ceb103dc28e383e514cd25f1799eac023d62c772549507867d5237cc7ca83237"}
May 20 15:32:28.006639 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:28.006595    2570 generic.go:358] "Generic (PLEG): container finished" podID="01c8fe5f-0c58-4e0b-b213-d815ec0448bd" containerID="a192f4f54fbabe3022fffd3da354fdad8d77af58c0a1127252398c7540f5ff77" exitCode=0
May 20 15:32:28.006639 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:28.006639    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerDied","Data":"a192f4f54fbabe3022fffd3da354fdad8d77af58c0a1127252398c7540f5ff77"}
May 20 15:32:28.755812 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:28.755597    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:28.755967 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:28.755597    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:28.755967 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:28.755869    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:28.755967 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:28.755904    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:29.011325 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:29.011244    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2b4js" event={"ID":"01c8fe5f-0c58-4e0b-b213-d815ec0448bd","Type":"ContainerStarted","Data":"fa4b7c56fa5ce904335c3e7ca5b7210ab81a196c182e93e47bb26e1adde36dec"}
May 20 15:32:29.033705 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:29.033658    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2b4js" podStartSLOduration=6.473910286 podStartE2EDuration="38.033644816s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:54.28625816 +0000 UTC m=+3.116354059" lastFinishedPulling="2026-05-20 15:32:25.845992691 +0000 UTC m=+34.676088589" observedRunningTime="2026-05-20 15:32:29.033399192 +0000 UTC m=+37.863495113" watchObservedRunningTime="2026-05-20 15:32:29.033644816 +0000 UTC m=+37.863740737"
May 20 15:32:30.755946 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:30.755902    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:30.756488 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:30.755902    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:30.756488 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:30.756043    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:30.756488 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:30.756071    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:32.754915 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:32.754873    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:32.755305 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:32.754873    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:32.755305 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:32.754984    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:32.755305 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:32.755066    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:34.755524 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:34.755472    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:34.755928 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:34.755472    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:34.755928 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:34.755600    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:34.755928 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:34.755668    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:36.755205 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:36.755176    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:36.755589 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:36.755176    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:36.755589 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:36.755282    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:36.755589 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:36.755379    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:38.755115 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:38.755076    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:38.755617 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:38.755190    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlb7f" podUID="f946a720-3300-4953-a930-3f99f4b9443c"
May 20 15:32:38.755617 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:38.755243    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:38.755617 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:38.755337    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fh2l" podUID="06532912-05d4-4680-bf74-819d7d4a36a0"
May 20 15:32:39.948848 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:39.948819    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeReady"
May 20 15:32:39.949218 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:39.948945    2570 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 15:32:40.028915 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.028880    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9t4sb"]
May 20 15:32:40.060278 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.060249    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kx27r"]
May 20 15:32:40.060446 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.060417    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9t4sb"
May 20 15:32:40.063457 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.063422    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 15:32:40.063788 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.063762    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 15:32:40.063919 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.063823    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zj5xk\""
May 20 15:32:40.063919 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.063823    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 15:32:40.074152 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.074129    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kx27r"]
May 20 15:32:40.074152 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.074154    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9t4sb"]
May 20 15:32:40.074304 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.074259    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.077825 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.077804    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kjpkx\""
May 20 15:32:40.077946 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.077863    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 15:32:40.078183 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.078163    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 15:32:40.078813 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.078798    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 15:32:40.078873 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.078802    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 15:32:40.091183 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.091152    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f26dc20-4712-42d6-8fd1-a60eca4d0f20-cert\") pod \"ingress-canary-9t4sb\" (UID: \"2f26dc20-4712-42d6-8fd1-a60eca4d0f20\") " pod="openshift-ingress-canary/ingress-canary-9t4sb"
May 20 15:32:40.091183 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.091185    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ds7h\" (UniqueName: \"kubernetes.io/projected/fd53f4ec-df9b-4e18-85c3-364ad87d120b-kube-api-access-7ds7h\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.091411 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.091208    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd53f4ec-df9b-4e18-85c3-364ad87d120b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.091411 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.091296    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd53f4ec-df9b-4e18-85c3-364ad87d120b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.091411 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.091335    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd53f4ec-df9b-4e18-85c3-364ad87d120b-crio-socket\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.091411 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.091378    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldblj\" (UniqueName: \"kubernetes.io/projected/2f26dc20-4712-42d6-8fd1-a60eca4d0f20-kube-api-access-ldblj\") pod \"ingress-canary-9t4sb\" (UID: \"2f26dc20-4712-42d6-8fd1-a60eca4d0f20\") " pod="openshift-ingress-canary/ingress-canary-9t4sb"
May 20 15:32:40.091411 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.091397    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd53f4ec-df9b-4e18-85c3-364ad87d120b-data-volume\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.134611 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.134578    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v6dcb"]
May 20 15:32:40.160332 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.160304    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.161128 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.161106    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v6dcb"]
May 20 15:32:40.164604 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.164583    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zhm2v\""
May 20 15:32:40.166223 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.166012    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 15:32:40.169659 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.169638    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 15:32:40.192402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192342    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd53f4ec-df9b-4e18-85c3-364ad87d120b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.192601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192412    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd53f4ec-df9b-4e18-85c3-364ad87d120b-crio-socket\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.192601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192439    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f09d029-10af-4b5f-b7b4-533f16160db1-metrics-tls\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.192601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192459    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldblj\" (UniqueName: \"kubernetes.io/projected/2f26dc20-4712-42d6-8fd1-a60eca4d0f20-kube-api-access-ldblj\") pod \"ingress-canary-9t4sb\" (UID: \"2f26dc20-4712-42d6-8fd1-a60eca4d0f20\") " pod="openshift-ingress-canary/ingress-canary-9t4sb"
May 20 15:32:40.192601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192479    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd53f4ec-df9b-4e18-85c3-364ad87d120b-data-volume\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.192601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192507    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f09d029-10af-4b5f-b7b4-533f16160db1-tmp-dir\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.192601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k7b8\" (UniqueName: \"kubernetes.io/projected/5f09d029-10af-4b5f-b7b4-533f16160db1-kube-api-access-6k7b8\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.192601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192584    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f26dc20-4712-42d6-8fd1-a60eca4d0f20-cert\") pod \"ingress-canary-9t4sb\" (UID: \"2f26dc20-4712-42d6-8fd1-a60eca4d0f20\") " pod="openshift-ingress-canary/ingress-canary-9t4sb"
May 20 15:32:40.192972 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192613    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7ds7h\" (UniqueName: \"kubernetes.io/projected/fd53f4ec-df9b-4e18-85c3-364ad87d120b-kube-api-access-7ds7h\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.192972 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192641    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f09d029-10af-4b5f-b7b4-533f16160db1-config-volume\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.192972 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192670    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd53f4ec-df9b-4e18-85c3-364ad87d120b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.192972 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192673    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd53f4ec-df9b-4e18-85c3-364ad87d120b-crio-socket\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.192972 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.192936    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd53f4ec-df9b-4e18-85c3-364ad87d120b-data-volume\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.193188 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.193169    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd53f4ec-df9b-4e18-85c3-364ad87d120b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.196541 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.196520    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd53f4ec-df9b-4e18-85c3-364ad87d120b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.196627 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.196540    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f26dc20-4712-42d6-8fd1-a60eca4d0f20-cert\") pod \"ingress-canary-9t4sb\" (UID: \"2f26dc20-4712-42d6-8fd1-a60eca4d0f20\") " pod="openshift-ingress-canary/ingress-canary-9t4sb"
May 20 15:32:40.203892 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.203826    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldblj\" (UniqueName: \"kubernetes.io/projected/2f26dc20-4712-42d6-8fd1-a60eca4d0f20-kube-api-access-ldblj\") pod \"ingress-canary-9t4sb\" (UID: \"2f26dc20-4712-42d6-8fd1-a60eca4d0f20\") " pod="openshift-ingress-canary/ingress-canary-9t4sb"
May 20 15:32:40.204559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.204539    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ds7h\" (UniqueName: \"kubernetes.io/projected/fd53f4ec-df9b-4e18-85c3-364ad87d120b-kube-api-access-7ds7h\") pod \"insights-runtime-extractor-kx27r\" (UID: \"fd53f4ec-df9b-4e18-85c3-364ad87d120b\") " pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.293795 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.293752    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f09d029-10af-4b5f-b7b4-533f16160db1-metrics-tls\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.293795 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.293793    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f09d029-10af-4b5f-b7b4-533f16160db1-tmp-dir\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.293991 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.293811    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6k7b8\" (UniqueName: \"kubernetes.io/projected/5f09d029-10af-4b5f-b7b4-533f16160db1-kube-api-access-6k7b8\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.293991 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.293852    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f09d029-10af-4b5f-b7b4-533f16160db1-config-volume\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.294142 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.294120    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f09d029-10af-4b5f-b7b4-533f16160db1-tmp-dir\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.294351 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.294326    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f09d029-10af-4b5f-b7b4-533f16160db1-config-volume\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.296217 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.296190    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f09d029-10af-4b5f-b7b4-533f16160db1-metrics-tls\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.308271 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.308241    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k7b8\" (UniqueName: \"kubernetes.io/projected/5f09d029-10af-4b5f-b7b4-533f16160db1-kube-api-access-6k7b8\") pod \"dns-default-v6dcb\" (UID: \"5f09d029-10af-4b5f-b7b4-533f16160db1\") " pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.370228 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.370198    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9t4sb"
May 20 15:32:40.382499 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.382470    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kx27r"
May 20 15:32:40.469974 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.469615    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:40.556434 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.556347    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kx27r"]
May 20 15:32:40.566389 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.566320    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9t4sb"]
May 20 15:32:40.586748 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:40.586113    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f26dc20_4712_42d6_8fd1_a60eca4d0f20.slice/crio-ebc8376de8c8aed99eb2d73372750d06fb7f25a9c9de511f5706c36b38a873c7 WatchSource:0}: Error finding container ebc8376de8c8aed99eb2d73372750d06fb7f25a9c9de511f5706c36b38a873c7: Status 404 returned error can't find the container with id ebc8376de8c8aed99eb2d73372750d06fb7f25a9c9de511f5706c36b38a873c7
May 20 15:32:40.660976 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.660946    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v6dcb"]
May 20 15:32:40.664482 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:40.664453    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f09d029_10af_4b5f_b7b4_533f16160db1.slice/crio-52e03a30bace1f2b84380dfb2fc5e610dfa08e7e95eff1e3c10ec9b430425407 WatchSource:0}: Error finding container 52e03a30bace1f2b84380dfb2fc5e610dfa08e7e95eff1e3c10ec9b430425407: Status 404 returned error can't find the container with id 52e03a30bace1f2b84380dfb2fc5e610dfa08e7e95eff1e3c10ec9b430425407
May 20 15:32:40.755803 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.755700    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:40.755956 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.755700    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:40.760073 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.760050    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-twbq4\""
May 20 15:32:40.760448 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.760428    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:32:40.760543 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.760453    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pdmqw\""
May 20 15:32:40.761179 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.761159    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:32:40.761273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.761254    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:32:40.983278 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.983243    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f6cf67b-dfxtr"]
May 20 15:32:40.996722 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:40.996690    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.001159 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.001129    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-service-ca\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.001324 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.001180    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9kjp\" (UniqueName: \"kubernetes.io/projected/c71935dc-7ff2-4b43-a695-4332ea7a3028-kube-api-access-j9kjp\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.001324 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.001239    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-serving-cert\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.001453 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.001331    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-oauth-config\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.001453 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.001399    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-config\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.001453 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.001430    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-oauth-serving-cert\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.008893 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.008830    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 20 15:32:41.010862 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.010836    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 20 15:32:41.028647 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.028611    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 20 15:32:41.028839 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.028612    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 20 15:32:41.028911 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.028849    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 20 15:32:41.028971 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.028961    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zdh57\""
May 20 15:32:41.029030 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.028963    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 20 15:32:41.029583 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.029557    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 20 15:32:41.031189 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.031168    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6cf67b-dfxtr"]
May 20 15:32:41.036117 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.036079    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6dcb" event={"ID":"5f09d029-10af-4b5f-b7b4-533f16160db1","Type":"ContainerStarted","Data":"52e03a30bace1f2b84380dfb2fc5e610dfa08e7e95eff1e3c10ec9b430425407"}
May 20 15:32:41.037739 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.037709    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kx27r" event={"ID":"fd53f4ec-df9b-4e18-85c3-364ad87d120b","Type":"ContainerStarted","Data":"d84d383edce32cf01b031c122fc82ab77413601d8205468d8f670633e0e6157a"}
May 20 15:32:41.037857 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.037747    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kx27r" event={"ID":"fd53f4ec-df9b-4e18-85c3-364ad87d120b","Type":"ContainerStarted","Data":"6e1b8496430aff8944a509032181ddfcb55f0823259f02c374db8e7530f13c0a"}
May 20 15:32:41.039040 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.039006    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9t4sb" event={"ID":"2f26dc20-4712-42d6-8fd1-a60eca4d0f20","Type":"ContainerStarted","Data":"ebc8376de8c8aed99eb2d73372750d06fb7f25a9c9de511f5706c36b38a873c7"}
May 20 15:32:41.102284 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.102238    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j9kjp\" (UniqueName: \"kubernetes.io/projected/c71935dc-7ff2-4b43-a695-4332ea7a3028-kube-api-access-j9kjp\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.102475 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.102319    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-serving-cert\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.102475 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.102388    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-oauth-config\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.102475 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.102430    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-config\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.102475 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.102456    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-oauth-serving-cert\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.102695 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.102487    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-service-ca\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.103740 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.103510    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-config\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.103740 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.103543    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-service-ca\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.103740 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.103681    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-oauth-serving-cert\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.105624 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.105600    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-oauth-config\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.105836 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.105791    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-serving-cert\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.114926 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.114895    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9kjp\" (UniqueName: \"kubernetes.io/projected/c71935dc-7ff2-4b43-a695-4332ea7a3028-kube-api-access-j9kjp\") pod \"console-5f6cf67b-dfxtr\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") " pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.307588 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.307547    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:41.543949 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.543891    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6cf67b-dfxtr"]
May 20 15:32:41.629995 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:41.629907    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71935dc_7ff2_4b43_a695_4332ea7a3028.slice/crio-52cd439101b382457bdd47523f10d09d194495307e45f6dce0b3999b1b6d77db WatchSource:0}: Error finding container 52cd439101b382457bdd47523f10d09d194495307e45f6dce0b3999b1b6d77db: Status 404 returned error can't find the container with id 52cd439101b382457bdd47523f10d09d194495307e45f6dce0b3999b1b6d77db
May 20 15:32:41.745054 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.745019    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"]
May 20 15:32:41.762092 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.762064    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"
May 20 15:32:41.765075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.765054    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-86qtg\""
May 20 15:32:41.765343 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.765323    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 20 15:32:41.775888 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.775864    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"]
May 20 15:32:41.808430 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.808387    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f9e81865-bde8-47dd-8c27-3c34161de4bf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd\" (UID: \"f9e81865-bde8-47dd-8c27-3c34161de4bf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"
May 20 15:32:41.908933 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:41.908851    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f9e81865-bde8-47dd-8c27-3c34161de4bf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd\" (UID: \"f9e81865-bde8-47dd-8c27-3c34161de4bf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"
May 20 15:32:41.909119 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:41.908995    2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 20 15:32:41.909119 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:41.909082    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e81865-bde8-47dd-8c27-3c34161de4bf-tls-certificates podName:f9e81865-bde8-47dd-8c27-3c34161de4bf nodeName:}" failed. No retries permitted until 2026-05-20 15:32:42.409064305 +0000 UTC m=+51.239160204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f9e81865-bde8-47dd-8c27-3c34161de4bf-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd" (UID: "f9e81865-bde8-47dd-8c27-3c34161de4bf") : secret "prometheus-operator-admission-webhook-tls" not found
May 20 15:32:42.042945 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:42.042909    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kx27r" event={"ID":"fd53f4ec-df9b-4e18-85c3-364ad87d120b","Type":"ContainerStarted","Data":"6fb5ffc56ae518112352ce65a7081588dbcb7236d959033af8cfcabe4f088355"}
May 20 15:32:42.044033 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:42.044003    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6cf67b-dfxtr" event={"ID":"c71935dc-7ff2-4b43-a695-4332ea7a3028","Type":"ContainerStarted","Data":"52cd439101b382457bdd47523f10d09d194495307e45f6dce0b3999b1b6d77db"}
May 20 15:32:42.413674 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:42.413632    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f9e81865-bde8-47dd-8c27-3c34161de4bf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd\" (UID: \"f9e81865-bde8-47dd-8c27-3c34161de4bf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"
May 20 15:32:42.416504 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:42.416475    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f9e81865-bde8-47dd-8c27-3c34161de4bf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd\" (UID: \"f9e81865-bde8-47dd-8c27-3c34161de4bf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"
May 20 15:32:42.674719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:42.674629    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"
May 20 15:32:43.038710 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:43.038658    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"]
May 20 15:32:43.042431 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:43.042393    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e81865_bde8_47dd_8c27_3c34161de4bf.slice/crio-ab58c3305b024dd0fe6977d69d8bc0c6b94b03d3504626524672241a3b07795f WatchSource:0}: Error finding container ab58c3305b024dd0fe6977d69d8bc0c6b94b03d3504626524672241a3b07795f: Status 404 returned error can't find the container with id ab58c3305b024dd0fe6977d69d8bc0c6b94b03d3504626524672241a3b07795f
May 20 15:32:43.047188 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:43.047152    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd" event={"ID":"f9e81865-bde8-47dd-8c27-3c34161de4bf","Type":"ContainerStarted","Data":"ab58c3305b024dd0fe6977d69d8bc0c6b94b03d3504626524672241a3b07795f"}
May 20 15:32:43.050315 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:43.050283    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9t4sb" event={"ID":"2f26dc20-4712-42d6-8fd1-a60eca4d0f20","Type":"ContainerStarted","Data":"e77cfb89f1e9ad5e2a18c17efc2a78b1c02124ae9ed385d8d4ef6c4932d1e2b6"}
May 20 15:32:43.052878 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:43.052852    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6dcb" event={"ID":"5f09d029-10af-4b5f-b7b4-533f16160db1","Type":"ContainerStarted","Data":"d55e670edc6d0951551a7bb458cced981789f990962b30d52ed63159fe6cfc50"}
May 20 15:32:43.094009 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:43.093949    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9t4sb" podStartSLOduration=0.810202112 podStartE2EDuration="3.093930089s" podCreationTimestamp="2026-05-20 15:32:40 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.589488555 +0000 UTC m=+49.419584467" lastFinishedPulling="2026-05-20 15:32:42.873216543 +0000 UTC m=+51.703312444" observedRunningTime="2026-05-20 15:32:43.091861125 +0000 UTC m=+51.921957047" watchObservedRunningTime="2026-05-20 15:32:43.093930089 +0000 UTC m=+51.924026005"
May 20 15:32:44.058438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:44.058380    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6dcb" event={"ID":"5f09d029-10af-4b5f-b7b4-533f16160db1","Type":"ContainerStarted","Data":"5c022d2ed72cc82566298e55ac044ebec54ed5070bd124a0b76be37aeef376fa"}
May 20 15:32:44.058873 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:44.058709    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:44.080522 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:44.080452    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-v6dcb" podStartSLOduration=1.867723703 podStartE2EDuration="4.080415009s" podCreationTimestamp="2026-05-20 15:32:40 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.66612834 +0000 UTC m=+49.496224239" lastFinishedPulling="2026-05-20 15:32:42.878819631 +0000 UTC m=+51.708915545" observedRunningTime="2026-05-20 15:32:44.079909526 +0000 UTC m=+52.910005447" watchObservedRunningTime="2026-05-20 15:32:44.080415009 +0000 UTC m=+52.910510927"
May 20 15:32:47.068632 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.068591    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kx27r" event={"ID":"fd53f4ec-df9b-4e18-85c3-364ad87d120b","Type":"ContainerStarted","Data":"6e3ae45280e9c93471dd8ce3ce31c7e191ddd8d1a589e2c98eda000ed91ea7a4"}
May 20 15:32:47.069976 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.069950    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd" event={"ID":"f9e81865-bde8-47dd-8c27-3c34161de4bf","Type":"ContainerStarted","Data":"57a879cedb48203ff1bd93a6f40e6ccd91229d4b47556db285fa9c3af4358fc2"}
May 20 15:32:47.070119 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.070059    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"
May 20 15:32:47.071381 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.071346    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6cf67b-dfxtr" event={"ID":"c71935dc-7ff2-4b43-a695-4332ea7a3028","Type":"ContainerStarted","Data":"45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608"}
May 20 15:32:47.075174 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.075151    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd"
May 20 15:32:47.087163 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.087113    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kx27r" podStartSLOduration=1.648483561 podStartE2EDuration="7.087098642s" podCreationTimestamp="2026-05-20 15:32:40 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.717231657 +0000 UTC m=+49.547327556" lastFinishedPulling="2026-05-20 15:32:46.155846723 +0000 UTC m=+54.985942637" observedRunningTime="2026-05-20 15:32:47.086630503 +0000 UTC m=+55.916726425" watchObservedRunningTime="2026-05-20 15:32:47.087098642 +0000 UTC m=+55.917194562"
May 20 15:32:47.104571 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.104516    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd" podStartSLOduration=2.994145677 podStartE2EDuration="6.104501852s" podCreationTimestamp="2026-05-20 15:32:41 +0000 UTC" firstStartedPulling="2026-05-20 15:32:43.045492206 +0000 UTC m=+51.875588110" lastFinishedPulling="2026-05-20 15:32:46.155848371 +0000 UTC m=+54.985944285" observedRunningTime="2026-05-20 15:32:47.103843334 +0000 UTC m=+55.933939256" watchObservedRunningTime="2026-05-20 15:32:47.104501852 +0000 UTC m=+55.934597773"
May 20 15:32:47.123800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.123740    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5f6cf67b-dfxtr" podStartSLOduration=2.603413144 podStartE2EDuration="7.123721589s" podCreationTimestamp="2026-05-20 15:32:40 +0000 UTC" firstStartedPulling="2026-05-20 15:32:41.642549061 +0000 UTC m=+50.472644960" lastFinishedPulling="2026-05-20 15:32:46.162857493 +0000 UTC m=+54.992953405" observedRunningTime="2026-05-20 15:32:47.12283878 +0000 UTC m=+55.952934712" watchObservedRunningTime="2026-05-20 15:32:47.123721589 +0000 UTC m=+55.953817512"
May 20 15:32:47.787722 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.787686    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"]
May 20 15:32:47.802897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.802866    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"]
May 20 15:32:47.803042 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.802992    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.805635 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.805609    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 20 15:32:47.805780 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.805656    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 20 15:32:47.805780 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.805661    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 15:32:47.806439 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.806353    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 15:32:47.806439 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.806356    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 15:32:47.806439 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.806394    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-854ww\""
May 20 15:32:47.863471 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.863431    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp844\" (UniqueName: \"kubernetes.io/projected/e9221be2-2270-403a-bf4d-5584c6301d71-kube-api-access-xp844\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.863649 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.863484    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9221be2-2270-403a-bf4d-5584c6301d71-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.863649 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.863519    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9221be2-2270-403a-bf4d-5584c6301d71-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.863649 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.863551    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9221be2-2270-403a-bf4d-5584c6301d71-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.964243 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.964207    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9221be2-2270-403a-bf4d-5584c6301d71-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.964360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.964258    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9221be2-2270-403a-bf4d-5584c6301d71-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.964360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.964295    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xp844\" (UniqueName: \"kubernetes.io/projected/e9221be2-2270-403a-bf4d-5584c6301d71-kube-api-access-xp844\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.964360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.964329    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9221be2-2270-403a-bf4d-5584c6301d71-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.964360 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:47.964352    2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 20 15:32:47.964659 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:47.964439    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9221be2-2270-403a-bf4d-5584c6301d71-prometheus-operator-tls podName:e9221be2-2270-403a-bf4d-5584c6301d71 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:48.464422175 +0000 UTC m=+57.294518074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/e9221be2-2270-403a-bf4d-5584c6301d71-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-6h7s6" (UID: "e9221be2-2270-403a-bf4d-5584c6301d71") : secret "prometheus-operator-tls" not found
May 20 15:32:47.964921 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.964902    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9221be2-2270-403a-bf4d-5584c6301d71-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.966846 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.966829    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9221be2-2270-403a-bf4d-5584c6301d71-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:47.975687 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:47.975659    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp844\" (UniqueName: \"kubernetes.io/projected/e9221be2-2270-403a-bf4d-5584c6301d71-kube-api-access-xp844\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:48.468695 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:48.468651    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9221be2-2270-403a-bf4d-5584c6301d71-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:48.471140 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:48.471116    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9221be2-2270-403a-bf4d-5584c6301d71-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-6h7s6\" (UID: \"e9221be2-2270-403a-bf4d-5584c6301d71\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:48.712010 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:48.711968    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"
May 20 15:32:48.838272 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:48.838237    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6"]
May 20 15:32:48.841514 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:48.841487    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9221be2_2270_403a_bf4d_5584c6301d71.slice/crio-d9526f754eb294d7401be838894fdc55662cf0e24a5eddc26ce7ec0fbbf7e465 WatchSource:0}: Error finding container d9526f754eb294d7401be838894fdc55662cf0e24a5eddc26ce7ec0fbbf7e465: Status 404 returned error can't find the container with id d9526f754eb294d7401be838894fdc55662cf0e24a5eddc26ce7ec0fbbf7e465
May 20 15:32:49.078463 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.078416    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6" event={"ID":"e9221be2-2270-403a-bf4d-5584c6301d71","Type":"ContainerStarted","Data":"d9526f754eb294d7401be838894fdc55662cf0e24a5eddc26ce7ec0fbbf7e465"}
May 20 15:32:49.305276 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.305237    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cd5bb9578-qplgg"]
May 20 15:32:49.307259 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.307227    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.316459 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.316284    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 20 15:32:49.320956 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.320913    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd5bb9578-qplgg"]
May 20 15:32:49.375611 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.375524    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-serving-cert\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.375611 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.375568    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-trusted-ca-bundle\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.375611 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.375592    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-oauth-serving-cert\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.375899 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.375674    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-console-config\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.375899 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.375760    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmp4\" (UniqueName: \"kubernetes.io/projected/45a886e9-a383-44f6-9b53-06981f870d27-kube-api-access-lrmp4\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.375899 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.375811    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-oauth-config\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.375899 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.375835    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-service-ca\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.477163 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.477129    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-serving-cert\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.477163 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.477169    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-trusted-ca-bundle\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.477694 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.477202    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-oauth-serving-cert\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.477694 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.477231    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-console-config\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.477694 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.477280    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmp4\" (UniqueName: \"kubernetes.io/projected/45a886e9-a383-44f6-9b53-06981f870d27-kube-api-access-lrmp4\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.477694 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.477471    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-oauth-config\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.477694 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.477506    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-service-ca\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.478110 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.478057    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-oauth-serving-cert\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.478397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.478355    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-console-config\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.479115 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.478643    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-trusted-ca-bundle\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.479115 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.479073    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-service-ca\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.480443 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.480412    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-serving-cert\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.480528 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.480465    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-oauth-config\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.487136 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.487104    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmp4\" (UniqueName: \"kubernetes.io/projected/45a886e9-a383-44f6-9b53-06981f870d27-kube-api-access-lrmp4\") pod \"console-6cd5bb9578-qplgg\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") " pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.620063 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.620012    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:32:49.955876 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.955723    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd5bb9578-qplgg"]
May 20 15:32:49.958920 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:49.958888    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a886e9_a383_44f6_9b53_06981f870d27.slice/crio-91bf85e252e590a821d18e3639b863eee02534b03bfe179ed2817a9ad78b3b0b WatchSource:0}: Error finding container 91bf85e252e590a821d18e3639b863eee02534b03bfe179ed2817a9ad78b3b0b: Status 404 returned error can't find the container with id 91bf85e252e590a821d18e3639b863eee02534b03bfe179ed2817a9ad78b3b0b
May 20 15:32:49.995958 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:49.995928    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw6k2"
May 20 15:32:50.086083 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:50.085986    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6" event={"ID":"e9221be2-2270-403a-bf4d-5584c6301d71","Type":"ContainerStarted","Data":"ecfd8d669f05113f84bdcb8c2b91d1625f69f858e9912c88fbd6d0d02d1eefe8"}
May 20 15:32:50.086083 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:50.086033    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6" event={"ID":"e9221be2-2270-403a-bf4d-5584c6301d71","Type":"ContainerStarted","Data":"9e7dfd1fa147835338f8aa959ff9523fa524c8feaf826cbff10c10463590889f"}
May 20 15:32:50.088469 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:50.088432    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd5bb9578-qplgg" event={"ID":"45a886e9-a383-44f6-9b53-06981f870d27","Type":"ContainerStarted","Data":"344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7"}
May 20 15:32:50.088642 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:50.088476    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd5bb9578-qplgg" event={"ID":"45a886e9-a383-44f6-9b53-06981f870d27","Type":"ContainerStarted","Data":"91bf85e252e590a821d18e3639b863eee02534b03bfe179ed2817a9ad78b3b0b"}
May 20 15:32:50.105912 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:50.105860    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-6h7s6" podStartSLOduration=2.09456886 podStartE2EDuration="3.105844281s" podCreationTimestamp="2026-05-20 15:32:47 +0000 UTC" firstStartedPulling="2026-05-20 15:32:48.843552789 +0000 UTC m=+57.673648701" lastFinishedPulling="2026-05-20 15:32:49.854828223 +0000 UTC m=+58.684924122" observedRunningTime="2026-05-20 15:32:50.104821772 +0000 UTC m=+58.934917690" watchObservedRunningTime="2026-05-20 15:32:50.105844281 +0000 UTC m=+58.935940198"
May 20 15:32:50.128381 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:50.128308    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6cd5bb9578-qplgg" podStartSLOduration=1.128293083 podStartE2EDuration="1.128293083s" podCreationTimestamp="2026-05-20 15:32:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:50.127940599 +0000 UTC m=+58.958036521" watchObservedRunningTime="2026-05-20 15:32:50.128293083 +0000 UTC m=+58.958389003"
May 20 15:32:51.308595 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:51.308559    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:51.308595 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:51.308605    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:51.313229 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:51.313205    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:52.097599 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.097567    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:32:52.160605 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.160576    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"]
May 20 15:32:52.163481 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.163465    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.166100 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.166068    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 20 15:32:52.166278 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.166256    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 20 15:32:52.166375 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.166312    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-znvbd\""
May 20 15:32:52.197065 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.197034    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"]
May 20 15:32:52.199083 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.199050    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.199256 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.199227    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.199331 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.199296    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcbf\" (UniqueName: \"kubernetes.io/projected/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-kube-api-access-zkcbf\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.199449 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.199430    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.218761 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.218730    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"]
May 20 15:32:52.220779 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.220764    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.223085 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.223059    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 20 15:32:52.223221 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.223142    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9fvf8\""
May 20 15:32:52.223352 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.223336    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 20 15:32:52.223442 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.223431    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 20 15:32:52.239469 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.239440    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"]
May 20 15:32:52.246904 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.246882    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d4vp5"]
May 20 15:32:52.248739 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.248723    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.251791 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.251771    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 15:32:52.251887 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.251808    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 15:32:52.252104 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.252089    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4w59b\""
May 20 15:32:52.252685 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.252669    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 15:32:52.300006 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.299979    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.300113 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.300113 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300065    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.300212 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300112    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/417faefd-8f15-4bd5-9974-563fbc982c35-root\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.300212 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:52.300116    2570 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 20 15:32:52.300212 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:52.300174    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-openshift-state-metrics-tls podName:2bddc1d4-016a-4456-b6d7-b2c8ec2a188b nodeName:}" failed. No retries permitted until 2026-05-20 15:32:52.800160338 +0000 UTC m=+61.630256237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-szwx5" (UID: "2bddc1d4-016a-4456-b6d7-b2c8ec2a188b") : secret "openshift-state-metrics-tls" not found
May 20 15:32:52.300326 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300259    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-wtmp\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.300326 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.300419 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300333    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.300419 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300355    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/417faefd-8f15-4bd5-9974-563fbc982c35-metrics-client-ca\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.300419 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300389    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3394f65c-c209-47f9-9af5-15b7e2db5592-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.300419 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300406    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.300601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300429    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcbf\" (UniqueName: \"kubernetes.io/projected/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-kube-api-access-zkcbf\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.300601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300446    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3394f65c-c209-47f9-9af5-15b7e2db5592-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.300601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300461    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/417faefd-8f15-4bd5-9974-563fbc982c35-sys\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.300601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300489    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-tls\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.300601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300560    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.300601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300591    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfzdq\" (UniqueName: \"kubernetes.io/projected/417faefd-8f15-4bd5-9974-563fbc982c35-kube-api-access-pfzdq\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.300834 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300620    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.300834 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300649    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-textfile\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.300834 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300687    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68f54\" (UniqueName: \"kubernetes.io/projected/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-api-access-68f54\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.300924 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.300868    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.302953 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.302932    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.329705 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.329677    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcbf\" (UniqueName: \"kubernetes.io/projected/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-kube-api-access-zkcbf\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.401849 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.401753    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-68f54\" (UniqueName: \"kubernetes.io/projected/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-api-access-68f54\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.401849 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.401827    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.401870    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.402062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.401897    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/417faefd-8f15-4bd5-9974-563fbc982c35-root\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.401926    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-wtmp\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.401962    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.402062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.401989    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/417faefd-8f15-4bd5-9974-563fbc982c35-metrics-client-ca\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402013    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3394f65c-c209-47f9-9af5-15b7e2db5592-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.402062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402038    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402074    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3394f65c-c209-47f9-9af5-15b7e2db5592-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.402397 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:52.402087    2570 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 20 15:32:52.402397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402102    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-wtmp\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402089    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/417faefd-8f15-4bd5-9974-563fbc982c35-root\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402397 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:52.402165    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-tls podName:3394f65c-c209-47f9-9af5-15b7e2db5592 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:52.902144041 +0000 UTC m=+61.732239954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-bch6c" (UID: "3394f65c-c209-47f9-9af5-15b7e2db5592") : secret "kube-state-metrics-tls" not found
May 20 15:32:52.402653 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402421    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/417faefd-8f15-4bd5-9974-563fbc982c35-sys\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402653 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402464    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-tls\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402653 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402505    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3394f65c-c209-47f9-9af5-15b7e2db5592-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.402653 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402540    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/417faefd-8f15-4bd5-9974-563fbc982c35-sys\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402653 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402580    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.402653 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:52.402597    2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 20 15:32:52.402929 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:52.402660    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-tls podName:417faefd-8f15-4bd5-9974-563fbc982c35 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:52.902641447 +0000 UTC m=+61.732737360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-tls") pod "node-exporter-d4vp5" (UID: "417faefd-8f15-4bd5-9974-563fbc982c35") : secret "node-exporter-tls" not found
May 20 15:32:52.402929 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402684    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pfzdq\" (UniqueName: \"kubernetes.io/projected/417faefd-8f15-4bd5-9974-563fbc982c35-kube-api-access-pfzdq\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402929 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402730    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-textfile\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402929 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402834    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3394f65c-c209-47f9-9af5-15b7e2db5592-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.402929 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402852    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.402929 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402863    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/417faefd-8f15-4bd5-9974-563fbc982c35-metrics-client-ca\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.403146 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.402965    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-textfile\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.403516 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.403495    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.404287 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.404268    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.404613 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.404595    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.410855 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.410830    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68f54\" (UniqueName: \"kubernetes.io/projected/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-api-access-68f54\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.411894 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.411866    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfzdq\" (UniqueName: \"kubernetes.io/projected/417faefd-8f15-4bd5-9974-563fbc982c35-kube-api-access-pfzdq\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.806037 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.806011    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.808332 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.808304    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bddc1d4-016a-4456-b6d7-b2c8ec2a188b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-szwx5\" (UID: \"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:52.907126 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.907095    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-tls\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.907273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.907181    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:52.909379 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.909341    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/417faefd-8f15-4bd5-9974-563fbc982c35-node-exporter-tls\") pod \"node-exporter-d4vp5\" (UID: \"417faefd-8f15-4bd5-9974-563fbc982c35\") " pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:52.909476 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:52.909380    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3394f65c-c209-47f9-9af5-15b7e2db5592-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-bch6c\" (UID: \"3394f65c-c209-47f9-9af5-15b7e2db5592\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:53.071860 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.071787    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"
May 20 15:32:53.129318 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.129277    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"
May 20 15:32:53.156704 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.156666    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d4vp5"
May 20 15:32:53.166933 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:53.166897    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417faefd_8f15_4bd5_9974_563fbc982c35.slice/crio-22c0874c7be63f05cea3731de8548fc0f3fa59974798e231aae1cbc9866c026c WatchSource:0}: Error finding container 22c0874c7be63f05cea3731de8548fc0f3fa59974798e231aae1cbc9866c026c: Status 404 returned error can't find the container with id 22c0874c7be63f05cea3731de8548fc0f3fa59974798e231aae1cbc9866c026c
May 20 15:32:53.219069 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.219004    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-szwx5"]
May 20 15:32:53.222554 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:53.222525    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bddc1d4_016a_4456_b6d7_b2c8ec2a188b.slice/crio-3ca0e289b25c79854cdf31070a27ee44c54948fa31ea2c02ccf4171ff15556d9 WatchSource:0}: Error finding container 3ca0e289b25c79854cdf31070a27ee44c54948fa31ea2c02ccf4171ff15556d9: Status 404 returned error can't find the container with id 3ca0e289b25c79854cdf31070a27ee44c54948fa31ea2c02ccf4171ff15556d9
May 20 15:32:53.277880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.277856    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c"]
May 20 15:32:53.280516 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:53.280489    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3394f65c_c209_47f9_9af5_15b7e2db5592.slice/crio-f8ee0d25749b89982c87b7c6fc5cbb287d4a93d04f2c6c99e8a8bd3e985f1a59 WatchSource:0}: Error finding container f8ee0d25749b89982c87b7c6fc5cbb287d4a93d04f2c6c99e8a8bd3e985f1a59: Status 404 returned error can't find the container with id f8ee0d25749b89982c87b7c6fc5cbb287d4a93d04f2c6c99e8a8bd3e985f1a59
May 20 15:32:53.343227 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.343188    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:32:53.345591 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.345567    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.349746 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.349720    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 20 15:32:53.349867 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.349766    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 20 15:32:53.349867 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.349720    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 20 15:32:53.349867 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.349821    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 20 15:32:53.349867 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.349720    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 20 15:32:53.350062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.349725    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 20 15:32:53.350062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.349719    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 20 15:32:53.350417 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.350399    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 20 15:32:53.350515 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.350417    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 20 15:32:53.351042 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.351025    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t5stx\""
May 20 15:32:53.370033 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.370000    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:32:53.412657 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412621    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-config-volume\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.412657 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412662    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.412902 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412721    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.412902 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412759    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtv4r\" (UniqueName: \"kubernetes.io/projected/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-kube-api-access-wtv4r\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.412902 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412785    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.412902 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412826    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-web-config\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.412902 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412847    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.412902 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412901    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.413111 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412920    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-config-out\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.413111 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.412978    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.413111 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.413003    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.413111 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.413021    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.413111 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.413060    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.513897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.513861    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514084 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.513908    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514084 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.513938    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514084 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.513982    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514084 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.514011    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-config-volume\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514084 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.514035    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514084 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.514072    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514405 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.514103    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wtv4r\" (UniqueName: \"kubernetes.io/projected/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-kube-api-access-wtv4r\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514405 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.514132    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514405 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.514179    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-web-config\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514732 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.514704    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.514917 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.514835    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.515028 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.514943    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.515212 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.515193    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.515343 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:53.515046    2570 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 20 15:32:53.515444 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:53.515434    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-main-tls podName:15b3f30b-8c95-4abc-bdb0-7adac4da8d44 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:54.015409413 +0000 UTC m=+62.845505325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "15b3f30b-8c95-4abc-bdb0-7adac4da8d44") : secret "alertmanager-main-tls" not found
May 20 15:32:53.515520 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.515331    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-config-out\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.515733 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.515708    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.517266 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.517239    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-web-config\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.518212 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.518164    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.518768 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.518696    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-config-out\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.518877 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.518859    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-config-volume\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.518944 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.518899    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.519084 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.519064    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.519314 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.519293    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.519891 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.519859    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.522667 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:53.522631    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtv4r\" (UniqueName: \"kubernetes.io/projected/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-kube-api-access-wtv4r\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:54.020844 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.020799    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:54.021009 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:54.020947    2570 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 20 15:32:54.021056 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:54.021037    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-main-tls podName:15b3f30b-8c95-4abc-bdb0-7adac4da8d44 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:55.021017797 +0000 UTC m=+63.851113707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "15b3f30b-8c95-4abc-bdb0-7adac4da8d44") : secret "alertmanager-main-tls" not found
May 20 15:32:54.064289 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.064260    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v6dcb"
May 20 15:32:54.101864 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.101811    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c" event={"ID":"3394f65c-c209-47f9-9af5-15b7e2db5592","Type":"ContainerStarted","Data":"f8ee0d25749b89982c87b7c6fc5cbb287d4a93d04f2c6c99e8a8bd3e985f1a59"}
May 20 15:32:54.104287 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.104208    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5" event={"ID":"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b","Type":"ContainerStarted","Data":"9636e12dc5296a1f5613ced3c9f1c29245b686d008de9f528507a4f65d895a29"}
May 20 15:32:54.104287 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.104253    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5" event={"ID":"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b","Type":"ContainerStarted","Data":"3e3949930833f1dc5fee02dc90875df15a72b14c431d527b61fb4dcb20c6e6bb"}
May 20 15:32:54.104287 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.104267    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5" event={"ID":"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b","Type":"ContainerStarted","Data":"3ca0e289b25c79854cdf31070a27ee44c54948fa31ea2c02ccf4171ff15556d9"}
May 20 15:32:54.105511 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.105472    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4vp5" event={"ID":"417faefd-8f15-4bd5-9974-563fbc982c35","Type":"ContainerStarted","Data":"22c0874c7be63f05cea3731de8548fc0f3fa59974798e231aae1cbc9866c026c"}
May 20 15:32:54.199477 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.199444    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-797ff79757-tvm9r"]
May 20 15:32:54.204342 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.204306    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.207325 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.207296    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 20 15:32:54.207664 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.207626    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 20 15:32:54.207664 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.207665    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 20 15:32:54.208144 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.208121    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 20 15:32:54.208474 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.208458    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 20 15:32:54.208773 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.208751    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x8nwl\""
May 20 15:32:54.208882 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.208754    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2gel3knfdnb9s\""
May 20 15:32:54.216433 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.216407    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-797ff79757-tvm9r"]
May 20 15:32:54.324062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.324025    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8lb6\" (UniqueName: \"kubernetes.io/projected/48466168-cf44-4695-acea-19a1139b9426-kube-api-access-h8lb6\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.324254 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.324071    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.324254 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.324093    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.324254 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.324114    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.324254 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.324195    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48466168-cf44-4695-acea-19a1139b9426-metrics-client-ca\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.324254 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.324238    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-tls\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.324254 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.324257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-grpc-tls\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.324529 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.324347    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.425720 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.425682    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-tls\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.425720 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.425722    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-grpc-tls\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.426192 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.425783    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.426192 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.425810    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h8lb6\" (UniqueName: \"kubernetes.io/projected/48466168-cf44-4695-acea-19a1139b9426-kube-api-access-h8lb6\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.426192 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.425836    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.426192 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.425854    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.426192 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.426005    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.426192 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.426105    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48466168-cf44-4695-acea-19a1139b9426-metrics-client-ca\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.426878 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.426854    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48466168-cf44-4695-acea-19a1139b9426-metrics-client-ca\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.428984 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.428955    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.429094 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.429019    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-grpc-tls\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.429094 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.429049    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.429174 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.429145    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.429212 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.429179    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-tls\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.429286 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.429268    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48466168-cf44-4695-acea-19a1139b9426-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.434122 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.434096    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8lb6\" (UniqueName: \"kubernetes.io/projected/48466168-cf44-4695-acea-19a1139b9426-kube-api-access-h8lb6\") pod \"thanos-querier-797ff79757-tvm9r\" (UID: \"48466168-cf44-4695-acea-19a1139b9426\") " pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:54.516232 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:54.516192    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:32:55.031931 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:55.031893    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:55.034470 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:55.034443    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15b3f30b-8c95-4abc-bdb0-7adac4da8d44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15b3f30b-8c95-4abc-bdb0-7adac4da8d44\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:55.158145 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:55.158104    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:55.451451 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:55.451205    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-797ff79757-tvm9r"]
May 20 15:32:55.454746 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:55.454698    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48466168_cf44_4695_acea_19a1139b9426.slice/crio-d534f07bef34f7f793ff11eaa83debd1d7be9eac9b0ada4c4965201f1069d0d2 WatchSource:0}: Error finding container d534f07bef34f7f793ff11eaa83debd1d7be9eac9b0ada4c4965201f1069d0d2: Status 404 returned error can't find the container with id d534f07bef34f7f793ff11eaa83debd1d7be9eac9b0ada4c4965201f1069d0d2
May 20 15:32:55.478704 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:55.478665    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:32:55.484192 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:55.483470    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b3f30b_8c95_4abc_bdb0_7adac4da8d44.slice/crio-35aa9439f39b84a3637a0d84d72c5ec5a14f37a1f3cae2c2247cd244fadda827 WatchSource:0}: Error finding container 35aa9439f39b84a3637a0d84d72c5ec5a14f37a1f3cae2c2247cd244fadda827: Status 404 returned error can't find the container with id 35aa9439f39b84a3637a0d84d72c5ec5a14f37a1f3cae2c2247cd244fadda827
May 20 15:32:56.116272 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.116163    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c" event={"ID":"3394f65c-c209-47f9-9af5-15b7e2db5592","Type":"ContainerStarted","Data":"6f5be3852830df339fda89f6ae2e7721ab7ffad14dad1403439d0a551b53d4fe"}
May 20 15:32:56.116272 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.116234    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c" event={"ID":"3394f65c-c209-47f9-9af5-15b7e2db5592","Type":"ContainerStarted","Data":"54361b5000be259e86e1ca43c424c41d9553f4b62fbf3ba20ffbbe42977a1db0"}
May 20 15:32:56.116272 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.116249    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c" event={"ID":"3394f65c-c209-47f9-9af5-15b7e2db5592","Type":"ContainerStarted","Data":"f60ed7f3deafef8b7499696e566c40247bd21b6265d1bb6358b0217d073af4e6"}
May 20 15:32:56.119433 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.119383    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5" event={"ID":"2bddc1d4-016a-4456-b6d7-b2c8ec2a188b","Type":"ContainerStarted","Data":"8921fe86b99ce644831542f62811b57b74620cd3bf92848e5a396639d76f4740"}
May 20 15:32:56.120639 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.120605    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r" event={"ID":"48466168-cf44-4695-acea-19a1139b9426","Type":"ContainerStarted","Data":"d534f07bef34f7f793ff11eaa83debd1d7be9eac9b0ada4c4965201f1069d0d2"}
May 20 15:32:56.122469 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.122436    2570 generic.go:358] "Generic (PLEG): container finished" podID="417faefd-8f15-4bd5-9974-563fbc982c35" containerID="430f32b4147d82f03fc8960b829285c1b16d01a27ad7a5ebe6eadee64879b3ea" exitCode=0
May 20 15:32:56.122582 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.122521    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4vp5" event={"ID":"417faefd-8f15-4bd5-9974-563fbc982c35","Type":"ContainerDied","Data":"430f32b4147d82f03fc8960b829285c1b16d01a27ad7a5ebe6eadee64879b3ea"}
May 20 15:32:56.124057 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.124018    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15b3f30b-8c95-4abc-bdb0-7adac4da8d44","Type":"ContainerStarted","Data":"35aa9439f39b84a3637a0d84d72c5ec5a14f37a1f3cae2c2247cd244fadda827"}
May 20 15:32:56.135307 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.135149    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bch6c" podStartSLOduration=2.153544636 podStartE2EDuration="4.135129392s" podCreationTimestamp="2026-05-20 15:32:52 +0000 UTC" firstStartedPulling="2026-05-20 15:32:53.283635583 +0000 UTC m=+62.113731489" lastFinishedPulling="2026-05-20 15:32:55.265220329 +0000 UTC m=+64.095316245" observedRunningTime="2026-05-20 15:32:56.133940921 +0000 UTC m=+64.964036866" watchObservedRunningTime="2026-05-20 15:32:56.135129392 +0000 UTC m=+64.965225304"
May 20 15:32:56.153053 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.152986    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-szwx5" podStartSLOduration=2.246490268 podStartE2EDuration="4.152965182s" podCreationTimestamp="2026-05-20 15:32:52 +0000 UTC" firstStartedPulling="2026-05-20 15:32:53.354300404 +0000 UTC m=+62.184396317" lastFinishedPulling="2026-05-20 15:32:55.260775315 +0000 UTC m=+64.090871231" observedRunningTime="2026-05-20 15:32:56.151435515 +0000 UTC m=+64.981531430" watchObservedRunningTime="2026-05-20 15:32:56.152965182 +0000 UTC m=+64.983061108"
May 20 15:32:56.623660 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.623624    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d6d988987-jx8z4"]
May 20 15:32:56.641046 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.640990    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d6d988987-jx8z4"]
May 20 15:32:56.641214 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.641125    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.643610 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.643585    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j67ml\""
May 20 15:32:56.643748 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.643683    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 20 15:32:56.643748 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.643692    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 20 15:32:56.643915 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.643899    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ddr5pvrpain4m\""
May 20 15:32:56.644401 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.644385    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 20 15:32:56.645175 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.645144    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 20 15:32:56.749421 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.749385    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a612e0d5-cf46-406e-83fe-448ed1d20967-metrics-server-audit-profiles\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.749719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.749446    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a612e0d5-cf46-406e-83fe-448ed1d20967-secret-metrics-server-client-certs\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.749719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.749471    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a612e0d5-cf46-406e-83fe-448ed1d20967-secret-metrics-server-tls\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.749719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.749498    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a612e0d5-cf46-406e-83fe-448ed1d20967-client-ca-bundle\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.749719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.749525    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a612e0d5-cf46-406e-83fe-448ed1d20967-audit-log\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.749719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.749604    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5t6w\" (UniqueName: \"kubernetes.io/projected/a612e0d5-cf46-406e-83fe-448ed1d20967-kube-api-access-t5t6w\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.749719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.749708    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a612e0d5-cf46-406e-83fe-448ed1d20967-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.850920 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.850885    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a612e0d5-cf46-406e-83fe-448ed1d20967-secret-metrics-server-client-certs\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.850920 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.850921    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a612e0d5-cf46-406e-83fe-448ed1d20967-secret-metrics-server-tls\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.851148 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.850940    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a612e0d5-cf46-406e-83fe-448ed1d20967-client-ca-bundle\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.851148 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.850962    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a612e0d5-cf46-406e-83fe-448ed1d20967-audit-log\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.851148 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.850990    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t5t6w\" (UniqueName: \"kubernetes.io/projected/a612e0d5-cf46-406e-83fe-448ed1d20967-kube-api-access-t5t6w\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.851148 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.851038    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a612e0d5-cf46-406e-83fe-448ed1d20967-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.851148 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.851117    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a612e0d5-cf46-406e-83fe-448ed1d20967-metrics-server-audit-profiles\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.851548 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.851521    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a612e0d5-cf46-406e-83fe-448ed1d20967-audit-log\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.852008 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.851979    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a612e0d5-cf46-406e-83fe-448ed1d20967-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.852132 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.852059    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a612e0d5-cf46-406e-83fe-448ed1d20967-metrics-server-audit-profiles\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.853708 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.853683    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a612e0d5-cf46-406e-83fe-448ed1d20967-secret-metrics-server-client-certs\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.853818 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.853792    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a612e0d5-cf46-406e-83fe-448ed1d20967-secret-metrics-server-tls\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.853939 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.853918    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a612e0d5-cf46-406e-83fe-448ed1d20967-client-ca-bundle\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.859066 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.859033    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5t6w\" (UniqueName: \"kubernetes.io/projected/a612e0d5-cf46-406e-83fe-448ed1d20967-kube-api-access-t5t6w\") pod \"metrics-server-7d6d988987-jx8z4\" (UID: \"a612e0d5-cf46-406e-83fe-448ed1d20967\") " pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.942382 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.942279    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd5bb9578-qplgg"]
May 20 15:32:56.953723 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.953677    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:32:56.955191 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.955170    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"]
May 20 15:32:56.985862 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.985792    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"]
May 20 15:32:56.985862 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.985829    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-56f5c86bdd-lkcv9"]
May 20 15:32:56.986049 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.985953    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"
May 20 15:32:56.988253 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.988185    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 20 15:32:56.988604 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:56.988561    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-hqvn8\""
May 20 15:32:57.014297 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.014260    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f5c86bdd-lkcv9"]
May 20 15:32:57.014464 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.014421    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.052606 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.052573    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/21208836-9aa6-4fb9-a7e6-8f15cd60fdb3-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-fhnzt\" (UID: \"21208836-9aa6-4fb9-a7e6-8f15cd60fdb3\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"
May 20 15:32:57.121687 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.121645    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d6d988987-jx8z4"]
May 20 15:32:57.128861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.128828    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4vp5" event={"ID":"417faefd-8f15-4bd5-9974-563fbc982c35","Type":"ContainerStarted","Data":"831c586a15de91486458396cbc53c56c4034a3cc20bf01b0c41941eec2fc5d34"}
May 20 15:32:57.129082 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.128869    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4vp5" event={"ID":"417faefd-8f15-4bd5-9974-563fbc982c35","Type":"ContainerStarted","Data":"57d134e1b8329f4cc2db2ad26ba88dafe63782607933a337a2f753bcf2ea31b1"}
May 20 15:32:57.130284 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.130252    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15b3f30b-8c95-4abc-bdb0-7adac4da8d44","Type":"ContainerStarted","Data":"5d0680e6ac071d6bd4f9703ecad432be3e5e1719bd9c5c7c67cb6fcf8d232097"}
May 20 15:32:57.149693 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.149465    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d4vp5" podStartSLOduration=3.058606666 podStartE2EDuration="5.149443709s" podCreationTimestamp="2026-05-20 15:32:52 +0000 UTC" firstStartedPulling="2026-05-20 15:32:53.169186253 +0000 UTC m=+61.999282165" lastFinishedPulling="2026-05-20 15:32:55.26002331 +0000 UTC m=+64.090119208" observedRunningTime="2026-05-20 15:32:57.149177263 +0000 UTC m=+65.979273210" watchObservedRunningTime="2026-05-20 15:32:57.149443709 +0000 UTC m=+65.979539631"
May 20 15:32:57.153654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.153614    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/21208836-9aa6-4fb9-a7e6-8f15cd60fdb3-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-fhnzt\" (UID: \"21208836-9aa6-4fb9-a7e6-8f15cd60fdb3\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"
May 20 15:32:57.153820 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.153667    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-trusted-ca-bundle\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.153820 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.153694    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-oauth-config\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.153820 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.153737    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/1d9bc028-587d-4213-a322-9ea0364a7109-kube-api-access-88q8z\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.153820 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.153787    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-console-config\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.153997 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.153836    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-oauth-serving-cert\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.153997 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.153899    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-serving-cert\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.154097 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.153990    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-service-ca\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.154097 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:57.154010    2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 20 15:32:57.154097 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:32:57.154084    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21208836-9aa6-4fb9-a7e6-8f15cd60fdb3-monitoring-plugin-cert podName:21208836-9aa6-4fb9-a7e6-8f15cd60fdb3 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:57.654061333 +0000 UTC m=+66.484157252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/21208836-9aa6-4fb9-a7e6-8f15cd60fdb3-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-fhnzt" (UID: "21208836-9aa6-4fb9-a7e6-8f15cd60fdb3") : secret "monitoring-plugin-cert" not found
May 20 15:32:57.255060 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.254943    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-trusted-ca-bundle\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.255060 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.255015    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-oauth-config\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.255303 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.255091    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/1d9bc028-587d-4213-a322-9ea0364a7109-kube-api-access-88q8z\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.255303 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.255162    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-console-config\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.255951 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.255924    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-oauth-serving-cert\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.256093 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.256060    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-serving-cert\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.256163 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.256118    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-service-ca\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.256511 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.256215    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-console-config\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.256629 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.256539    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-oauth-serving-cert\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.256780 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.256733    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-trusted-ca-bundle\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.257214 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.257193    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-service-ca\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.258851 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.258829    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-serving-cert\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.259076 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.259051    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-oauth-config\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.263350 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.263330    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/1d9bc028-587d-4213-a322-9ea0364a7109-kube-api-access-88q8z\") pod \"console-56f5c86bdd-lkcv9\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") " pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.324590 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.324546    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:32:57.559971 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.559931    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:57.560151 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.560003    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:57.562459 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.562434    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:32:57.562601 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.562479    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:32:57.572799 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.572769    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06532912-05d4-4680-bf74-819d7d4a36a0-metrics-certs\") pod \"network-metrics-daemon-6fh2l\" (UID: \"06532912-05d4-4680-bf74-819d7d4a36a0\") " pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:57.573013 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.572995    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:32:57.583734 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.583705    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnld9\" (UniqueName: \"kubernetes.io/projected/f946a720-3300-4953-a930-3f99f4b9443c-kube-api-access-pnld9\") pod \"network-check-target-nlb7f\" (UID: \"f946a720-3300-4953-a930-3f99f4b9443c\") " pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:57.661463 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.661419    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/21208836-9aa6-4fb9-a7e6-8f15cd60fdb3-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-fhnzt\" (UID: \"21208836-9aa6-4fb9-a7e6-8f15cd60fdb3\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"
May 20 15:32:57.664483 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.664457    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/21208836-9aa6-4fb9-a7e6-8f15cd60fdb3-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-fhnzt\" (UID: \"21208836-9aa6-4fb9-a7e6-8f15cd60fdb3\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"
May 20 15:32:57.679111 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:57.679062    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda612e0d5_cf46_406e_83fe_448ed1d20967.slice/crio-eec0095b06bcd3e219acf3de9ec9fb0592307eceaa52787d4cc005f26284b2b3 WatchSource:0}: Error finding container eec0095b06bcd3e219acf3de9ec9fb0592307eceaa52787d4cc005f26284b2b3: Status 404 returned error can't find the container with id eec0095b06bcd3e219acf3de9ec9fb0592307eceaa52787d4cc005f26284b2b3
May 20 15:32:57.815863 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.815829    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f5c86bdd-lkcv9"]
May 20 15:32:57.819449 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:57.819412    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9bc028_587d_4213_a322_9ea0364a7109.slice/crio-640734b058e283867998f79b34755084248c2b54bf80a5924d8f301d615b2bb6 WatchSource:0}: Error finding container 640734b058e283867998f79b34755084248c2b54bf80a5924d8f301d615b2bb6: Status 404 returned error can't find the container with id 640734b058e283867998f79b34755084248c2b54bf80a5924d8f301d615b2bb6
May 20 15:32:57.867540 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.867512    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pdmqw\""
May 20 15:32:57.871901 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.871796    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-twbq4\""
May 20 15:32:57.875722 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.875702    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:32:57.880332 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.880309    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fh2l"
May 20 15:32:57.906488 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:57.906429    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"
May 20 15:32:58.042573 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.042510    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nlb7f"]
May 20 15:32:58.047398 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:58.047291    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf946a720_3300_4953_a930_3f99f4b9443c.slice/crio-4968d5775c2310468be58c0c9e73a7941b804292f7760deeb0f36e0df768bf3f WatchSource:0}: Error finding container 4968d5775c2310468be58c0c9e73a7941b804292f7760deeb0f36e0df768bf3f: Status 404 returned error can't find the container with id 4968d5775c2310468be58c0c9e73a7941b804292f7760deeb0f36e0df768bf3f
May 20 15:32:58.065761 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.065664    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6fh2l"]
May 20 15:32:58.069690 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:58.069644    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06532912_05d4_4680_bf74_819d7d4a36a0.slice/crio-2e0c46d7365341120f422a4ef1453cdd275137f92493157614a0377464c9a55d WatchSource:0}: Error finding container 2e0c46d7365341120f422a4ef1453cdd275137f92493157614a0377464c9a55d: Status 404 returned error can't find the container with id 2e0c46d7365341120f422a4ef1453cdd275137f92493157614a0377464c9a55d
May 20 15:32:58.088775 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.088737    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"]
May 20 15:32:58.094230 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:32:58.094187    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21208836_9aa6_4fb9_a7e6_8f15cd60fdb3.slice/crio-66558f8a743ddfa6765d5126970fad5baf0ca073b2e0f18e2f0248c07bcfa5df WatchSource:0}: Error finding container 66558f8a743ddfa6765d5126970fad5baf0ca073b2e0f18e2f0248c07bcfa5df: Status 404 returned error can't find the container with id 66558f8a743ddfa6765d5126970fad5baf0ca073b2e0f18e2f0248c07bcfa5df
May 20 15:32:58.135761 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.135712    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f5c86bdd-lkcv9" event={"ID":"1d9bc028-587d-4213-a322-9ea0364a7109","Type":"ContainerStarted","Data":"c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325"}
May 20 15:32:58.135761 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.135767    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f5c86bdd-lkcv9" event={"ID":"1d9bc028-587d-4213-a322-9ea0364a7109","Type":"ContainerStarted","Data":"640734b058e283867998f79b34755084248c2b54bf80a5924d8f301d615b2bb6"}
May 20 15:32:58.140262 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.140226    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r" event={"ID":"48466168-cf44-4695-acea-19a1139b9426","Type":"ContainerStarted","Data":"dcac042f0782942329eccf3c1a1d2aa6f5008ba091a6afc9907a9efd61c5a5d5"}
May 20 15:32:58.140460 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.140269    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r" event={"ID":"48466168-cf44-4695-acea-19a1139b9426","Type":"ContainerStarted","Data":"848b9d3b8cd41a073317219d4631d37483b1f2e15541519fc4e8748af068245e"}
May 20 15:32:58.140460 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.140283    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r" event={"ID":"48466168-cf44-4695-acea-19a1139b9426","Type":"ContainerStarted","Data":"acdd70a3f4ba49aaeb4883b34ff69e5337de2ae71f7772cc22ce1a8f9aec8489"}
May 20 15:32:58.141648 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.141617    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fh2l" event={"ID":"06532912-05d4-4680-bf74-819d7d4a36a0","Type":"ContainerStarted","Data":"2e0c46d7365341120f422a4ef1453cdd275137f92493157614a0377464c9a55d"}
May 20 15:32:58.143009 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.142969    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nlb7f" event={"ID":"f946a720-3300-4953-a930-3f99f4b9443c","Type":"ContainerStarted","Data":"4968d5775c2310468be58c0c9e73a7941b804292f7760deeb0f36e0df768bf3f"}
May 20 15:32:58.144430 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.144401    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4" event={"ID":"a612e0d5-cf46-406e-83fe-448ed1d20967","Type":"ContainerStarted","Data":"eec0095b06bcd3e219acf3de9ec9fb0592307eceaa52787d4cc005f26284b2b3"}
May 20 15:32:58.145961 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.145932    2570 generic.go:358] "Generic (PLEG): container finished" podID="15b3f30b-8c95-4abc-bdb0-7adac4da8d44" containerID="5d0680e6ac071d6bd4f9703ecad432be3e5e1719bd9c5c7c67cb6fcf8d232097" exitCode=0
May 20 15:32:58.146097 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.146008    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15b3f30b-8c95-4abc-bdb0-7adac4da8d44","Type":"ContainerDied","Data":"5d0680e6ac071d6bd4f9703ecad432be3e5e1719bd9c5c7c67cb6fcf8d232097"}
May 20 15:32:58.147457 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.147392    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt" event={"ID":"21208836-9aa6-4fb9-a7e6-8f15cd60fdb3","Type":"ContainerStarted","Data":"66558f8a743ddfa6765d5126970fad5baf0ca073b2e0f18e2f0248c07bcfa5df"}
May 20 15:32:58.153743 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.153690    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-56f5c86bdd-lkcv9" podStartSLOduration=2.153673455 podStartE2EDuration="2.153673455s" podCreationTimestamp="2026-05-20 15:32:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:58.15337833 +0000 UTC m=+66.983474245" watchObservedRunningTime="2026-05-20 15:32:58.153673455 +0000 UTC m=+66.983769377"
May 20 15:32:58.476313 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.475079    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 15:32:58.479559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.479523    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.487603 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.486666    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 20 15:32:58.487603 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.486986    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 20 15:32:58.487603 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.487187    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 20 15:32:58.492120 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.492099    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 20 15:32:58.492345 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.492329    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 20 15:32:58.494459 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.492769    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 20 15:32:58.494459 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.492947    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 20 15:32:58.494459 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.493025    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 20 15:32:58.494459 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.493173    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 20 15:32:58.494459 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.493260    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5zn5n\""
May 20 15:32:58.494459 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.493408    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 20 15:32:58.494459 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.494038    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-81hth5sh25ffi\""
May 20 15:32:58.498378 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.498032    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 20 15:32:58.509425 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.509221    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 20 15:32:58.515714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.515674    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 15:32:58.574597 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574554    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.574872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574626    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-config-out\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.574872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574656    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.574872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574686    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.574872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574726    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.574872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574755    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.574872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574792    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8htj\" (UniqueName: \"kubernetes.io/projected/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-kube-api-access-c8htj\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.574872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574825    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.574872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574864    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.575744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.575744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574937    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.575744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574964    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-config\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.575744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.574998    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.575744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.575037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.575744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.575087    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.575744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.575110    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-web-config\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.575744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.575150    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.575744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.575176    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.675960    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676009    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676045    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676090    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-config-out\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676117    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676144    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676180    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676205    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676239    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c8htj\" (UniqueName: \"kubernetes.io/projected/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-kube-api-access-c8htj\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676271    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676309    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676355    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676396    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676424    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-config\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676456    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676488    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.677397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676530    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.678582 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.676555    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-web-config\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.684633 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.680592    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.684633 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.681190    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.684633 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.682068    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.684633 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.684139    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.686056 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.685752    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.686056 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.685793    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.688408 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.687949    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.688408 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.688342    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-web-config\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.688865 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.688838    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.689924 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.689899    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-config-out\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.693383 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.691819    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8htj\" (UniqueName: \"kubernetes.io/projected/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-kube-api-access-c8htj\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.693383 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.691875    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.693383 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.692225    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-config\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.693383 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.692634    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.693383 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.693091    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.693383 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.693159    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.693717 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.693700    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.696867 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.696838    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/daa09e04-7ad0-42f0-ac58-c98dd9e63c4c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:58.803444 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:58.803133    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:32:59.621199 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:32:59.621154    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:33:01.686260 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:01.681337    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 15:33:01.867965 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:33:01.867920    2570 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa09e04_7ad0_42f0_ac58_c98dd9e63c4c.slice/crio-conmon-86f8a14549734f8ee784bc836a397b19e475275abc6d334b124f996de8b1c729.scope\": RecentStats: unable to find data in memory cache]"
May 20 15:33:02.166421 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.166360    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r" event={"ID":"48466168-cf44-4695-acea-19a1139b9426","Type":"ContainerStarted","Data":"3b5cb5ec2a67104b44de2aae68897d38c68943dbf7540283a02d4279efcdfe58"}
May 20 15:33:02.166421 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.166421    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r" event={"ID":"48466168-cf44-4695-acea-19a1139b9426","Type":"ContainerStarted","Data":"818c3e328265317ad45599ef2914e1e8b798f1f7edb73aa06f0b0ab4f2e65614"}
May 20 15:33:02.166682 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.166436    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r" event={"ID":"48466168-cf44-4695-acea-19a1139b9426","Type":"ContainerStarted","Data":"7c9e933ce97dc266252a6c0f9a8f700151b21a270b56895fd45f7a1eb938edd7"}
May 20 15:33:02.166843 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.166817    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:33:02.168503 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.168469    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fh2l" event={"ID":"06532912-05d4-4680-bf74-819d7d4a36a0","Type":"ContainerStarted","Data":"2b6e0863d498bac0366aa0bf2ae70adafd3cd3f7e94f560cec1d31c07be82251"}
May 20 15:33:02.168638 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.168507    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fh2l" event={"ID":"06532912-05d4-4680-bf74-819d7d4a36a0","Type":"ContainerStarted","Data":"cdd37b7afa1ed169a1f5b18dd4bb22cd08c2d498f8517988c6ea40b940816688"}
May 20 15:33:02.169943 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.169910    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4" event={"ID":"a612e0d5-cf46-406e-83fe-448ed1d20967","Type":"ContainerStarted","Data":"4ff7fcc0e92c50d71be7b865464f66875c0e1d71813e361bdeeff34a240ad014"}
May 20 15:33:02.171554 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.171527    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nlb7f" event={"ID":"f946a720-3300-4953-a930-3f99f4b9443c","Type":"ContainerStarted","Data":"f4056fbcbf66850d3264c0301c993dcadcc9925b86839e4f84636022b6dc499c"}
May 20 15:33:02.171662 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.171622    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:33:02.174900 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.174858    2570 generic.go:358] "Generic (PLEG): container finished" podID="daa09e04-7ad0-42f0-ac58-c98dd9e63c4c" containerID="86f8a14549734f8ee784bc836a397b19e475275abc6d334b124f996de8b1c729" exitCode=0
May 20 15:33:02.175072 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.174948    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c","Type":"ContainerDied","Data":"86f8a14549734f8ee784bc836a397b19e475275abc6d334b124f996de8b1c729"}
May 20 15:33:02.175072 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.174991    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c","Type":"ContainerStarted","Data":"91e5236919eef3dd61c09b023ccef8b1451bd49875e0c65ede05385b0140ab61"}
May 20 15:33:02.175178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.175146    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r"
May 20 15:33:02.178434 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.178297    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15b3f30b-8c95-4abc-bdb0-7adac4da8d44","Type":"ContainerStarted","Data":"7f39a2f90ae775ae303ce3466e2547c5eb69f91c0ae491a57fdf4c29d1c088c4"}
May 20 15:33:02.178434 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.178332    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15b3f30b-8c95-4abc-bdb0-7adac4da8d44","Type":"ContainerStarted","Data":"e4a9b326ff1d6fb3aae018a7e55d273f17efbbfa08a47a965dbf71e009365c85"}
May 20 15:33:02.178434 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.178344    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15b3f30b-8c95-4abc-bdb0-7adac4da8d44","Type":"ContainerStarted","Data":"21d899e67e886e8b7676fd8cdca5f310056f1c0128319abbe05876b527b32589"}
May 20 15:33:02.178434 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.178388    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15b3f30b-8c95-4abc-bdb0-7adac4da8d44","Type":"ContainerStarted","Data":"4e95cddbe46a36279ff43da7f61e27314bdc422ff49cbfb16b92cfdfdd30431c"}
May 20 15:33:02.178434 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.178400    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15b3f30b-8c95-4abc-bdb0-7adac4da8d44","Type":"ContainerStarted","Data":"354430d456c8d90e77780d5795ed8dc0eff144997f99144bdbddfa78d0af5012"}
May 20 15:33:02.178434 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.178412    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15b3f30b-8c95-4abc-bdb0-7adac4da8d44","Type":"ContainerStarted","Data":"84f543ddcba51521fc87bbbd2b06cd5a50b90f9132a02434edea69362b1576ef"}
May 20 15:33:02.179739 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.179717    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt" event={"ID":"21208836-9aa6-4fb9-a7e6-8f15cd60fdb3","Type":"ContainerStarted","Data":"48a3b796f993dd88c9895b0248928a9d5a71018eb33f3ddf0ebed11335314471"}
May 20 15:33:02.179965 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.179938    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"
May 20 15:33:02.185339 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.185316    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt"
May 20 15:33:02.194679 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.194625    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-797ff79757-tvm9r" podStartSLOduration=2.148639495 podStartE2EDuration="8.194609026s" podCreationTimestamp="2026-05-20 15:32:54 +0000 UTC" firstStartedPulling="2026-05-20 15:32:55.457537045 +0000 UTC m=+64.287632952" lastFinishedPulling="2026-05-20 15:33:01.503506572 +0000 UTC m=+70.333602483" observedRunningTime="2026-05-20 15:33:02.192884948 +0000 UTC m=+71.022980899" watchObservedRunningTime="2026-05-20 15:33:02.194609026 +0000 UTC m=+71.024704946"
May 20 15:33:02.268754 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.268637    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6fh2l" podStartSLOduration=66.832716379 podStartE2EDuration="1m10.268620675s" podCreationTimestamp="2026-05-20 15:31:52 +0000 UTC" firstStartedPulling="2026-05-20 15:32:58.072043716 +0000 UTC m=+66.902139633" lastFinishedPulling="2026-05-20 15:33:01.507948023 +0000 UTC m=+70.338043929" observedRunningTime="2026-05-20 15:33:02.267027393 +0000 UTC m=+71.097123315" watchObservedRunningTime="2026-05-20 15:33:02.268620675 +0000 UTC m=+71.098716596"
May 20 15:33:02.283127 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.283063    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-fhnzt" podStartSLOduration=2.874641483 podStartE2EDuration="6.283035209s" podCreationTimestamp="2026-05-20 15:32:56 +0000 UTC" firstStartedPulling="2026-05-20 15:32:58.096808622 +0000 UTC m=+66.926904530" lastFinishedPulling="2026-05-20 15:33:01.50520234 +0000 UTC m=+70.335298256" observedRunningTime="2026-05-20 15:33:02.282416644 +0000 UTC m=+71.112512585" watchObservedRunningTime="2026-05-20 15:33:02.283035209 +0000 UTC m=+71.113131131"
May 20 15:33:02.314051 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.313996    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.294976345 podStartE2EDuration="9.313973962s" podCreationTimestamp="2026-05-20 15:32:53 +0000 UTC" firstStartedPulling="2026-05-20 15:32:55.485709599 +0000 UTC m=+64.315805498" lastFinishedPulling="2026-05-20 15:33:01.504707199 +0000 UTC m=+70.334803115" observedRunningTime="2026-05-20 15:33:02.313338181 +0000 UTC m=+71.143434097" watchObservedRunningTime="2026-05-20 15:33:02.313973962 +0000 UTC m=+71.144069886"
May 20 15:33:02.335166 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.335096    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nlb7f" podStartSLOduration=67.835408531 podStartE2EDuration="1m11.335073998s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:32:58.050212368 +0000 UTC m=+66.880308267" lastFinishedPulling="2026-05-20 15:33:01.549877821 +0000 UTC m=+70.379973734" observedRunningTime="2026-05-20 15:33:02.33264519 +0000 UTC m=+71.162741114" watchObservedRunningTime="2026-05-20 15:33:02.335073998 +0000 UTC m=+71.165169920"
May 20 15:33:02.357295 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:02.357218    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4" podStartSLOduration=2.5362470200000002 podStartE2EDuration="6.35720019s" podCreationTimestamp="2026-05-20 15:32:56 +0000 UTC" firstStartedPulling="2026-05-20 15:32:57.683433029 +0000 UTC m=+66.513528941" lastFinishedPulling="2026-05-20 15:33:01.504386189 +0000 UTC m=+70.334482111" observedRunningTime="2026-05-20 15:33:02.356000054 +0000 UTC m=+71.186096004" watchObservedRunningTime="2026-05-20 15:33:02.35720019 +0000 UTC m=+71.187296109"
May 20 15:33:05.194805 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:05.194715    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c","Type":"ContainerStarted","Data":"40c51f535f632e7ed8c0f6d86121306de3fc43230553eacd060e19c4070a2d13"}
May 20 15:33:05.194805 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:05.194762    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c","Type":"ContainerStarted","Data":"f6f5b25a33d03f6a3312b69ad1e0b988bc85f5ea11963ca2bd761fef7e37d7e1"}
May 20 15:33:05.194805 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:05.194776    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c","Type":"ContainerStarted","Data":"f58045a3b04d61b4df42bae92100717628883a698d7ef9c9e2d26f4ddaad0d59"}
May 20 15:33:05.194805 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:05.194788    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c","Type":"ContainerStarted","Data":"040d013152535084b18e80671b65ac7287c1e58fcf14b679dfb8ab319be34dcf"}
May 20 15:33:06.200430 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:06.200392    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c","Type":"ContainerStarted","Data":"d352b160fa6895e680c861e5acb12fefb84604c59cfd7469e5e790cd1b7567ea"}
May 20 15:33:06.200430 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:06.200429    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"daa09e04-7ad0-42f0-ac58-c98dd9e63c4c","Type":"ContainerStarted","Data":"bba75827f69b14eaa3d11d25a22c1f83fff726f9cf96a01a46571881ebe7ff9b"}
May 20 15:33:07.324869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:07.324827    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:33:07.324869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:07.324878    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:33:07.329796 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:07.329769    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:33:07.348646 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:07.348604    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.724999553 podStartE2EDuration="9.34859099s" podCreationTimestamp="2026-05-20 15:32:58 +0000 UTC" firstStartedPulling="2026-05-20 15:33:02.176261365 +0000 UTC m=+71.006357277" lastFinishedPulling="2026-05-20 15:33:04.799852805 +0000 UTC m=+73.629948714" observedRunningTime="2026-05-20 15:33:06.231834969 +0000 UTC m=+75.061930955" watchObservedRunningTime="2026-05-20 15:33:07.34859099 +0000 UTC m=+76.178686910"
May 20 15:33:08.211288 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:08.211260    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:33:08.304880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:08.304842    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6cf67b-dfxtr"]
May 20 15:33:08.804606 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:08.804556    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:33:16.954648 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:16.954612    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:33:16.955002 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:16.954660    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:33:21.963040 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:21.962995    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6cd5bb9578-qplgg" podUID="45a886e9-a383-44f6-9b53-06981f870d27" containerName="console" containerID="cri-o://344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7" gracePeriod=15
May 20 15:33:22.213604 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.213550    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6cd5bb9578-qplgg_45a886e9-a383-44f6-9b53-06981f870d27/console/0.log"
May 20 15:33:22.213733 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.213621    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:33:22.261428 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.261403    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6cd5bb9578-qplgg_45a886e9-a383-44f6-9b53-06981f870d27/console/0.log"
May 20 15:33:22.261543 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.261440    2570 generic.go:358] "Generic (PLEG): container finished" podID="45a886e9-a383-44f6-9b53-06981f870d27" containerID="344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7" exitCode=2
May 20 15:33:22.261543 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.261480    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd5bb9578-qplgg" event={"ID":"45a886e9-a383-44f6-9b53-06981f870d27","Type":"ContainerDied","Data":"344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7"}
May 20 15:33:22.261543 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.261505    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd5bb9578-qplgg" event={"ID":"45a886e9-a383-44f6-9b53-06981f870d27","Type":"ContainerDied","Data":"91bf85e252e590a821d18e3639b863eee02534b03bfe179ed2817a9ad78b3b0b"}
May 20 15:33:22.261543 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.261511    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd5bb9578-qplgg"
May 20 15:33:22.261543 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.261535    2570 scope.go:117] "RemoveContainer" containerID="344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7"
May 20 15:33:22.269157 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.269139    2570 scope.go:117] "RemoveContainer" containerID="344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7"
May 20 15:33:22.269418 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:33:22.269394    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7\": container with ID starting with 344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7 not found: ID does not exist" containerID="344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7"
May 20 15:33:22.269512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.269423    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7"} err="failed to get container status \"344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7\": rpc error: code = NotFound desc = could not find container \"344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7\": container with ID starting with 344ae20e4ef39fc19fde8a6c8b58acd2ca5cbf7ada17753ff561c477f41fa1a7 not found: ID does not exist"
May 20 15:33:22.301764 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.301742    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-service-ca\") pod \"45a886e9-a383-44f6-9b53-06981f870d27\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") "
May 20 15:33:22.301849 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.301801    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-oauth-serving-cert\") pod \"45a886e9-a383-44f6-9b53-06981f870d27\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") "
May 20 15:33:22.301849 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.301825    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-console-config\") pod \"45a886e9-a383-44f6-9b53-06981f870d27\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") "
May 20 15:33:22.301928 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.301849    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-oauth-config\") pod \"45a886e9-a383-44f6-9b53-06981f870d27\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") "
May 20 15:33:22.301928 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.301878    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-serving-cert\") pod \"45a886e9-a383-44f6-9b53-06981f870d27\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") "
May 20 15:33:22.301928 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.301923    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-trusted-ca-bundle\") pod \"45a886e9-a383-44f6-9b53-06981f870d27\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") "
May 20 15:33:22.302074 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.301977    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrmp4\" (UniqueName: \"kubernetes.io/projected/45a886e9-a383-44f6-9b53-06981f870d27-kube-api-access-lrmp4\") pod \"45a886e9-a383-44f6-9b53-06981f870d27\" (UID: \"45a886e9-a383-44f6-9b53-06981f870d27\") "
May 20 15:33:22.302180 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.302154    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-service-ca" (OuterVolumeSpecName: "service-ca") pod "45a886e9-a383-44f6-9b53-06981f870d27" (UID: "45a886e9-a383-44f6-9b53-06981f870d27"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:22.302262 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.302216    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-console-config" (OuterVolumeSpecName: "console-config") pod "45a886e9-a383-44f6-9b53-06981f870d27" (UID: "45a886e9-a383-44f6-9b53-06981f870d27"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:22.302262 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.302230    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "45a886e9-a383-44f6-9b53-06981f870d27" (UID: "45a886e9-a383-44f6-9b53-06981f870d27"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:22.302341 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.302265    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:22.302570 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.302547    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "45a886e9-a383-44f6-9b53-06981f870d27" (UID: "45a886e9-a383-44f6-9b53-06981f870d27"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:22.304137 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.304107    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a886e9-a383-44f6-9b53-06981f870d27-kube-api-access-lrmp4" (OuterVolumeSpecName: "kube-api-access-lrmp4") pod "45a886e9-a383-44f6-9b53-06981f870d27" (UID: "45a886e9-a383-44f6-9b53-06981f870d27"). InnerVolumeSpecName "kube-api-access-lrmp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:33:22.304316 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.304290    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "45a886e9-a383-44f6-9b53-06981f870d27" (UID: "45a886e9-a383-44f6-9b53-06981f870d27"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:22.304438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.304387    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "45a886e9-a383-44f6-9b53-06981f870d27" (UID: "45a886e9-a383-44f6-9b53-06981f870d27"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:22.403063 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.403037    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:22.403063 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.403060    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:22.403063 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.403070    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:22.403253 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.403078    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45a886e9-a383-44f6-9b53-06981f870d27-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:22.403253 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.403094    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a886e9-a383-44f6-9b53-06981f870d27-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:22.403253 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.403103    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrmp4\" (UniqueName: \"kubernetes.io/projected/45a886e9-a383-44f6-9b53-06981f870d27-kube-api-access-lrmp4\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:22.583591 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.583560    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd5bb9578-qplgg"]
May 20 15:33:22.587710 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:22.587678    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cd5bb9578-qplgg"]
May 20 15:33:23.759893 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:23.759857    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45a886e9-a383-44f6-9b53-06981f870d27" path="/var/lib/kubelet/pods/45a886e9-a383-44f6-9b53-06981f870d27/volumes"
May 20 15:33:33.187134 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.187104    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nlb7f"
May 20 15:33:33.325251 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.325197    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5f6cf67b-dfxtr" podUID="c71935dc-7ff2-4b43-a695-4332ea7a3028" containerName="console" containerID="cri-o://45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608" gracePeriod=15
May 20 15:33:33.616001 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.615982    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f6cf67b-dfxtr_c71935dc-7ff2-4b43-a695-4332ea7a3028/console/0.log"
May 20 15:33:33.616112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.616039    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:33:33.702292 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702263    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-oauth-config\") pod \"c71935dc-7ff2-4b43-a695-4332ea7a3028\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") "
May 20 15:33:33.702458 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702306    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-serving-cert\") pod \"c71935dc-7ff2-4b43-a695-4332ea7a3028\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") "
May 20 15:33:33.702458 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702323    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-config\") pod \"c71935dc-7ff2-4b43-a695-4332ea7a3028\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") "
May 20 15:33:33.702529 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702464    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9kjp\" (UniqueName: \"kubernetes.io/projected/c71935dc-7ff2-4b43-a695-4332ea7a3028-kube-api-access-j9kjp\") pod \"c71935dc-7ff2-4b43-a695-4332ea7a3028\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") "
May 20 15:33:33.702529 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702513    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-oauth-serving-cert\") pod \"c71935dc-7ff2-4b43-a695-4332ea7a3028\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") "
May 20 15:33:33.702612 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702561    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-service-ca\") pod \"c71935dc-7ff2-4b43-a695-4332ea7a3028\" (UID: \"c71935dc-7ff2-4b43-a695-4332ea7a3028\") "
May 20 15:33:33.702697 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702676    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-config" (OuterVolumeSpecName: "console-config") pod "c71935dc-7ff2-4b43-a695-4332ea7a3028" (UID: "c71935dc-7ff2-4b43-a695-4332ea7a3028"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:33.702886 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702864    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:33.703028 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702920    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-service-ca" (OuterVolumeSpecName: "service-ca") pod "c71935dc-7ff2-4b43-a695-4332ea7a3028" (UID: "c71935dc-7ff2-4b43-a695-4332ea7a3028"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:33.703028 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.702951    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c71935dc-7ff2-4b43-a695-4332ea7a3028" (UID: "c71935dc-7ff2-4b43-a695-4332ea7a3028"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:33.704519 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.704500    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c71935dc-7ff2-4b43-a695-4332ea7a3028" (UID: "c71935dc-7ff2-4b43-a695-4332ea7a3028"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:33.704949 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.704922    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c71935dc-7ff2-4b43-a695-4332ea7a3028" (UID: "c71935dc-7ff2-4b43-a695-4332ea7a3028"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:33.704949 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.704937    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71935dc-7ff2-4b43-a695-4332ea7a3028-kube-api-access-j9kjp" (OuterVolumeSpecName: "kube-api-access-j9kjp") pod "c71935dc-7ff2-4b43-a695-4332ea7a3028" (UID: "c71935dc-7ff2-4b43-a695-4332ea7a3028"). InnerVolumeSpecName "kube-api-access-j9kjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:33:33.803407 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.803378    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:33.803407 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.803403    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c71935dc-7ff2-4b43-a695-4332ea7a3028-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:33.803557 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.803413    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9kjp\" (UniqueName: \"kubernetes.io/projected/c71935dc-7ff2-4b43-a695-4332ea7a3028-kube-api-access-j9kjp\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:33.803557 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.803422    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:33.803557 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:33.803432    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c71935dc-7ff2-4b43-a695-4332ea7a3028-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:33:34.297166 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.297139    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f6cf67b-dfxtr_c71935dc-7ff2-4b43-a695-4332ea7a3028/console/0.log"
May 20 15:33:34.297579 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.297180    2570 generic.go:358] "Generic (PLEG): container finished" podID="c71935dc-7ff2-4b43-a695-4332ea7a3028" containerID="45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608" exitCode=2
May 20 15:33:34.297579 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.297221    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6cf67b-dfxtr" event={"ID":"c71935dc-7ff2-4b43-a695-4332ea7a3028","Type":"ContainerDied","Data":"45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608"}
May 20 15:33:34.297579 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.297252    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6cf67b-dfxtr"
May 20 15:33:34.297579 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.297263    2570 scope.go:117] "RemoveContainer" containerID="45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608"
May 20 15:33:34.297579 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.297251    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6cf67b-dfxtr" event={"ID":"c71935dc-7ff2-4b43-a695-4332ea7a3028","Type":"ContainerDied","Data":"52cd439101b382457bdd47523f10d09d194495307e45f6dce0b3999b1b6d77db"}
May 20 15:33:34.312349 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.312224    2570 scope.go:117] "RemoveContainer" containerID="45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608"
May 20 15:33:34.312595 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:33:34.312535    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608\": container with ID starting with 45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608 not found: ID does not exist" containerID="45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608"
May 20 15:33:34.312595 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.312569    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608"} err="failed to get container status \"45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608\": rpc error: code = NotFound desc = could not find container \"45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608\": container with ID starting with 45af003c918c3d834dc373c12a66cd37342eda48e2ecc1dc4584a4d1aca87608 not found: ID does not exist"
May 20 15:33:34.314260 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.314234    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6cf67b-dfxtr"]
May 20 15:33:34.318260 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:34.318237    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f6cf67b-dfxtr"]
May 20 15:33:35.759417 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:35.759382    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71935dc-7ff2-4b43-a695-4332ea7a3028" path="/var/lib/kubelet/pods/c71935dc-7ff2-4b43-a695-4332ea7a3028/volumes"
May 20 15:33:36.959706 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:36.959674    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:33:36.979570 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:36.979538    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d6d988987-jx8z4"
May 20 15:33:58.803891 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:58.803847    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:33:58.823505 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:58.823478    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:33:59.387512 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:33:59.387484    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 15:34:16.537939 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.537895    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"]
May 20 15:34:16.538360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.538235    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a886e9-a383-44f6-9b53-06981f870d27" containerName="console"
May 20 15:34:16.538360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.538249    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a886e9-a383-44f6-9b53-06981f870d27" containerName="console"
May 20 15:34:16.538360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.538267    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71935dc-7ff2-4b43-a695-4332ea7a3028" containerName="console"
May 20 15:34:16.538360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.538275    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71935dc-7ff2-4b43-a695-4332ea7a3028" containerName="console"
May 20 15:34:16.538360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.538340    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="c71935dc-7ff2-4b43-a695-4332ea7a3028" containerName="console"
May 20 15:34:16.538360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.538348    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="45a886e9-a383-44f6-9b53-06981f870d27" containerName="console"
May 20 15:34:16.540507 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.540491    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.544189 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.543454    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 20 15:34:16.544336 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.544299    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 20 15:34:16.544336 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.544321    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 20 15:34:16.544772 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.544747    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 20 15:34:16.544843 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.544794    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 20 15:34:16.544944 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.544913    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-w2zbn\""
May 20 15:34:16.549104 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.548762    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 20 15:34:16.559056 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.559014    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"]
May 20 15:34:16.598587 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.598550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-secret-telemeter-client\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.598587 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.598592    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.598851 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.598617    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-telemeter-client-tls\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.598851 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.598720    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6274f8d-c395-4e58-9b3c-511f763a4869-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.598851 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.598769    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6274f8d-c395-4e58-9b3c-511f763a4869-metrics-client-ca\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.598965 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.598879    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-federate-client-tls\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.598965 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.598919    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6274f8d-c395-4e58-9b3c-511f763a4869-serving-certs-ca-bundle\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.599046 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.598997    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzpc\" (UniqueName: \"kubernetes.io/projected/f6274f8d-c395-4e58-9b3c-511f763a4869-kube-api-access-gmzpc\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.699698 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.699649    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzpc\" (UniqueName: \"kubernetes.io/projected/f6274f8d-c395-4e58-9b3c-511f763a4869-kube-api-access-gmzpc\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.699897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.699713    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-secret-telemeter-client\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.699897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.699741    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.699897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.699778    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-telemeter-client-tls\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.699897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.699809    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6274f8d-c395-4e58-9b3c-511f763a4869-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.699897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.699846    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6274f8d-c395-4e58-9b3c-511f763a4869-metrics-client-ca\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.700177 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.699898    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-federate-client-tls\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.700177 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.699925    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6274f8d-c395-4e58-9b3c-511f763a4869-serving-certs-ca-bundle\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.701041 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.701001    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6274f8d-c395-4e58-9b3c-511f763a4869-metrics-client-ca\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.701267 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.701227    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6274f8d-c395-4e58-9b3c-511f763a4869-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.701504 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.701481    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6274f8d-c395-4e58-9b3c-511f763a4869-serving-certs-ca-bundle\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.702786 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.702750    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-secret-telemeter-client\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.702786 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.702781    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-federate-client-tls\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.702964 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.702881    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.702964 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.702930    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6274f8d-c395-4e58-9b3c-511f763a4869-telemeter-client-tls\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.708498 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.708470    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzpc\" (UniqueName: \"kubernetes.io/projected/f6274f8d-c395-4e58-9b3c-511f763a4869-kube-api-access-gmzpc\") pod \"telemeter-client-55c5c6c85c-7qrft\" (UID: \"f6274f8d-c395-4e58-9b3c-511f763a4869\") " pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.851874 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.851775    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"
May 20 15:34:16.991435 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:16.991403    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55c5c6c85c-7qrft"]
May 20 15:34:16.994625 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:34:16.994585    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6274f8d_c395_4e58_9b3c_511f763a4869.slice/crio-123e6d03ab30bc43ac7f13a92f6fe8ef888ca862f43dd1a621b431d1dcef7b67 WatchSource:0}: Error finding container 123e6d03ab30bc43ac7f13a92f6fe8ef888ca862f43dd1a621b431d1dcef7b67: Status 404 returned error can't find the container with id 123e6d03ab30bc43ac7f13a92f6fe8ef888ca862f43dd1a621b431d1dcef7b67
May 20 15:34:17.426816 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:17.426780    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft" event={"ID":"f6274f8d-c395-4e58-9b3c-511f763a4869","Type":"ContainerStarted","Data":"123e6d03ab30bc43ac7f13a92f6fe8ef888ca862f43dd1a621b431d1dcef7b67"}
May 20 15:34:19.435523 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:19.435484    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft" event={"ID":"f6274f8d-c395-4e58-9b3c-511f763a4869","Type":"ContainerStarted","Data":"77a8a1d05359443b055d04e929c3bdf5d68ec1a845af9699834082562dc1d153"}
May 20 15:34:19.435523 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:19.435524    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft" event={"ID":"f6274f8d-c395-4e58-9b3c-511f763a4869","Type":"ContainerStarted","Data":"cc4140e7c393ac5fd1480122dd963917d10e057632c9e718d3524ae92a7160fc"}
May 20 15:34:19.435523 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:19.435533    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft" event={"ID":"f6274f8d-c395-4e58-9b3c-511f763a4869","Type":"ContainerStarted","Data":"897a14bf1f1cb1ae787799b162c910a95c071fd06e96550b55a94a6fce1e6f36"}
May 20 15:34:19.459163 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:19.459075    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55c5c6c85c-7qrft" podStartSLOduration=1.829088562 podStartE2EDuration="3.459053949s" podCreationTimestamp="2026-05-20 15:34:16 +0000 UTC" firstStartedPulling="2026-05-20 15:34:16.996553047 +0000 UTC m=+145.826648946" lastFinishedPulling="2026-05-20 15:34:18.62651843 +0000 UTC m=+147.456614333" observedRunningTime="2026-05-20 15:34:19.457734731 +0000 UTC m=+148.287830671" watchObservedRunningTime="2026-05-20 15:34:19.459053949 +0000 UTC m=+148.289149896"
May 20 15:34:20.244643 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.244604    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff5947f47-xbm4d"]
May 20 15:34:20.246849 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.246830    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.260108 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.260076    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff5947f47-xbm4d"]
May 20 15:34:20.334605 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.334563    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-console-config\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.334776 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.334638    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-serving-cert\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.334776 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.334668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-service-ca\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.334776 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.334690    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-trusted-ca-bundle\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.334776 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.334715    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-oauth-serving-cert\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.334776 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.334746    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tldqr\" (UniqueName: \"kubernetes.io/projected/e4958e27-4626-4556-b172-5ae410bd0daa-kube-api-access-tldqr\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.334776 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.334767    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-oauth-config\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.435765 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.435718    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-serving-cert\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.436248 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.435789    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-service-ca\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.436248 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.435826    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-trusted-ca-bundle\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.436248 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.435850    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-oauth-serving-cert\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.436248 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.435900    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tldqr\" (UniqueName: \"kubernetes.io/projected/e4958e27-4626-4556-b172-5ae410bd0daa-kube-api-access-tldqr\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.436248 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.435934    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-oauth-config\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.436248 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.435984    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-console-config\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.436822 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.436793    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-console-config\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.436911 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.436817    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-service-ca\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.436977 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.436943    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-oauth-serving-cert\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.437319 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.437294    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-trusted-ca-bundle\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.438578 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.438559    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-oauth-config\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.438675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.438647    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-serving-cert\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.444409 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.444386    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tldqr\" (UniqueName: \"kubernetes.io/projected/e4958e27-4626-4556-b172-5ae410bd0daa-kube-api-access-tldqr\") pod \"console-5ff5947f47-xbm4d\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") " pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.557022 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.556975    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:20.690095 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:20.690067    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff5947f47-xbm4d"]
May 20 15:34:20.692074 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:34:20.692030    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4958e27_4626_4556_b172_5ae410bd0daa.slice/crio-59a55589f4ed298a80f241bc389abf13c6b75ef33e2a2a2a038f49832cecbd4e WatchSource:0}: Error finding container 59a55589f4ed298a80f241bc389abf13c6b75ef33e2a2a2a038f49832cecbd4e: Status 404 returned error can't find the container with id 59a55589f4ed298a80f241bc389abf13c6b75ef33e2a2a2a038f49832cecbd4e
May 20 15:34:21.443334 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:21.443298    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff5947f47-xbm4d" event={"ID":"e4958e27-4626-4556-b172-5ae410bd0daa","Type":"ContainerStarted","Data":"801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed"}
May 20 15:34:21.443334 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:21.443334    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff5947f47-xbm4d" event={"ID":"e4958e27-4626-4556-b172-5ae410bd0daa","Type":"ContainerStarted","Data":"59a55589f4ed298a80f241bc389abf13c6b75ef33e2a2a2a038f49832cecbd4e"}
May 20 15:34:21.466014 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:21.465961    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5ff5947f47-xbm4d" podStartSLOduration=1.465946342 podStartE2EDuration="1.465946342s" podCreationTimestamp="2026-05-20 15:34:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:34:21.464540572 +0000 UTC m=+150.294636494" watchObservedRunningTime="2026-05-20 15:34:21.465946342 +0000 UTC m=+150.296042262"
May 20 15:34:30.557288 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:30.557241    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:30.557288 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:30.557295    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:30.562423 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:30.562393    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:31.479225 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.479193    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:34:31.532773 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.532736    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f5c86bdd-lkcv9"]
May 20 15:34:31.612615 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.612570    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bf6459878-zhk9m"]
May 20 15:34:31.615215 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.615187    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.626786 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.626759    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf6459878-zhk9m"]
May 20 15:34:31.742525 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.742435    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-oauth-serving-cert\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.742525 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.742470    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpqpn\" (UniqueName: \"kubernetes.io/projected/10d66096-0b67-4cbd-be29-2c08f88d145e-kube-api-access-lpqpn\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.742525 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.742489    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-trusted-ca-bundle\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.742738 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.742610    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-oauth-config\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.742738 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.742692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-serving-cert\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.742738 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.742711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-console-config\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.742738 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.742733    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-service-ca\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.843260 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.843212    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-serving-cert\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.843260 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.843262    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-console-config\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.843560 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.843291    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-service-ca\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.843560 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.843335    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-oauth-serving-cert\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.843560 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.843355    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lpqpn\" (UniqueName: \"kubernetes.io/projected/10d66096-0b67-4cbd-be29-2c08f88d145e-kube-api-access-lpqpn\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.843560 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.843409    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-trusted-ca-bundle\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.843560 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.843472    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-oauth-config\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.844150 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.844120    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-console-config\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.844275 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.844148    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-oauth-serving-cert\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.844275 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.844187    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-service-ca\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.844405 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.844294    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-trusted-ca-bundle\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.845866 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.845833    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-serving-cert\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.846002 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.845983    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-oauth-config\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.852969 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.852947    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpqpn\" (UniqueName: \"kubernetes.io/projected/10d66096-0b67-4cbd-be29-2c08f88d145e-kube-api-access-lpqpn\") pod \"console-5bf6459878-zhk9m\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") " pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:31.925995 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:31.925955    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:32.064036 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:32.063993    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf6459878-zhk9m"]
May 20 15:34:32.068342 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:34:32.068307    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d66096_0b67_4cbd_be29_2c08f88d145e.slice/crio-5887dae25918b8b97a7e07f76520eeeeb2f4723806d84677bce809d23c06b3fd WatchSource:0}: Error finding container 5887dae25918b8b97a7e07f76520eeeeb2f4723806d84677bce809d23c06b3fd: Status 404 returned error can't find the container with id 5887dae25918b8b97a7e07f76520eeeeb2f4723806d84677bce809d23c06b3fd
May 20 15:34:32.480374 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:32.480323    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf6459878-zhk9m" event={"ID":"10d66096-0b67-4cbd-be29-2c08f88d145e","Type":"ContainerStarted","Data":"467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1"}
May 20 15:34:32.480574 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:32.480380    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf6459878-zhk9m" event={"ID":"10d66096-0b67-4cbd-be29-2c08f88d145e","Type":"ContainerStarted","Data":"5887dae25918b8b97a7e07f76520eeeeb2f4723806d84677bce809d23c06b3fd"}
May 20 15:34:32.499610 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:32.499559    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5bf6459878-zhk9m" podStartSLOduration=1.49954003 podStartE2EDuration="1.49954003s" podCreationTimestamp="2026-05-20 15:34:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:34:32.498346312 +0000 UTC m=+161.328442234" watchObservedRunningTime="2026-05-20 15:34:32.49954003 +0000 UTC m=+161.329635954"
May 20 15:34:41.926533 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:41.926435    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:41.926533 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:41.926493    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:41.931586 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:41.931558    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:42.516634 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:42.516606    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:34:42.570287 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:42.570248    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff5947f47-xbm4d"]
May 20 15:34:56.552899 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.552849    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-56f5c86bdd-lkcv9" podUID="1d9bc028-587d-4213-a322-9ea0364a7109" containerName="console" containerID="cri-o://c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325" gracePeriod=15
May 20 15:34:56.789455 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.789431    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56f5c86bdd-lkcv9_1d9bc028-587d-4213-a322-9ea0364a7109/console/0.log"
May 20 15:34:56.789575 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.789491    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:34:56.862912 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.862831    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-service-ca\") pod \"1d9bc028-587d-4213-a322-9ea0364a7109\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") "
May 20 15:34:56.862912 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.862888    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-oauth-serving-cert\") pod \"1d9bc028-587d-4213-a322-9ea0364a7109\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") "
May 20 15:34:56.863072 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.862922    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-oauth-config\") pod \"1d9bc028-587d-4213-a322-9ea0364a7109\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") "
May 20 15:34:56.863106 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.863078    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-serving-cert\") pod \"1d9bc028-587d-4213-a322-9ea0364a7109\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") "
May 20 15:34:56.863139 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.863122    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-console-config\") pod \"1d9bc028-587d-4213-a322-9ea0364a7109\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") "
May 20 15:34:56.863175 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.863155    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/1d9bc028-587d-4213-a322-9ea0364a7109-kube-api-access-88q8z\") pod \"1d9bc028-587d-4213-a322-9ea0364a7109\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") "
May 20 15:34:56.863221 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.863208    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-trusted-ca-bundle\") pod \"1d9bc028-587d-4213-a322-9ea0364a7109\" (UID: \"1d9bc028-587d-4213-a322-9ea0364a7109\") "
May 20 15:34:56.863587 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.863561    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1d9bc028-587d-4213-a322-9ea0364a7109" (UID: "1d9bc028-587d-4213-a322-9ea0364a7109"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:34:56.863682 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.863555    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-service-ca" (OuterVolumeSpecName: "service-ca") pod "1d9bc028-587d-4213-a322-9ea0364a7109" (UID: "1d9bc028-587d-4213-a322-9ea0364a7109"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:34:56.864009 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.863963    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-console-config" (OuterVolumeSpecName: "console-config") pod "1d9bc028-587d-4213-a322-9ea0364a7109" (UID: "1d9bc028-587d-4213-a322-9ea0364a7109"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:34:56.864134 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.864041    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:34:56.864134 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.864079    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:34:56.865387 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.864516    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1d9bc028-587d-4213-a322-9ea0364a7109" (UID: "1d9bc028-587d-4213-a322-9ea0364a7109"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:34:56.866022 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.865991    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1d9bc028-587d-4213-a322-9ea0364a7109" (UID: "1d9bc028-587d-4213-a322-9ea0364a7109"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:56.868907 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.867586    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1d9bc028-587d-4213-a322-9ea0364a7109" (UID: "1d9bc028-587d-4213-a322-9ea0364a7109"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:56.870269 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.870121    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9bc028-587d-4213-a322-9ea0364a7109-kube-api-access-88q8z" (OuterVolumeSpecName: "kube-api-access-88q8z") pod "1d9bc028-587d-4213-a322-9ea0364a7109" (UID: "1d9bc028-587d-4213-a322-9ea0364a7109"). InnerVolumeSpecName "kube-api-access-88q8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:34:56.965466 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.965442    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:34:56.965611 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.965463    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d9bc028-587d-4213-a322-9ea0364a7109-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:34:56.965611 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.965487    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:34:56.965611 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.965497    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/1d9bc028-587d-4213-a322-9ea0364a7109-kube-api-access-88q8z\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:34:56.965611 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:56.965506    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9bc028-587d-4213-a322-9ea0364a7109-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:34:57.559814 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.559786    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56f5c86bdd-lkcv9_1d9bc028-587d-4213-a322-9ea0364a7109/console/0.log"
May 20 15:34:57.560246 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.559829    2570 generic.go:358] "Generic (PLEG): container finished" podID="1d9bc028-587d-4213-a322-9ea0364a7109" containerID="c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325" exitCode=2
May 20 15:34:57.560246 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.559896    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f5c86bdd-lkcv9" event={"ID":"1d9bc028-587d-4213-a322-9ea0364a7109","Type":"ContainerDied","Data":"c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325"}
May 20 15:34:57.560246 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.559916    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f5c86bdd-lkcv9"
May 20 15:34:57.560246 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.559933    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f5c86bdd-lkcv9" event={"ID":"1d9bc028-587d-4213-a322-9ea0364a7109","Type":"ContainerDied","Data":"640734b058e283867998f79b34755084248c2b54bf80a5924d8f301d615b2bb6"}
May 20 15:34:57.560246 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.559956    2570 scope.go:117] "RemoveContainer" containerID="c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325"
May 20 15:34:57.568807 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.568785    2570 scope.go:117] "RemoveContainer" containerID="c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325"
May 20 15:34:57.569040 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:34:57.569019    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325\": container with ID starting with c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325 not found: ID does not exist" containerID="c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325"
May 20 15:34:57.569096 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.569048    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325"} err="failed to get container status \"c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325\": rpc error: code = NotFound desc = could not find container \"c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325\": container with ID starting with c3e17d213e0058516a5e3595cc938635f9918ef3882491965e64bccf76ff8325 not found: ID does not exist"
May 20 15:34:57.582972 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.582951    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f5c86bdd-lkcv9"]
May 20 15:34:57.589580 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.589558    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56f5c86bdd-lkcv9"]
May 20 15:34:57.759007 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:34:57.758971    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d9bc028-587d-4213-a322-9ea0364a7109" path="/var/lib/kubelet/pods/1d9bc028-587d-4213-a322-9ea0364a7109/volumes"
May 20 15:35:04.774664 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.774627    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qs4db"]
May 20 15:35:04.775129 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.774965    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9bc028-587d-4213-a322-9ea0364a7109" containerName="console"
May 20 15:35:04.775129 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.774976    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9bc028-587d-4213-a322-9ea0364a7109" containerName="console"
May 20 15:35:04.775129 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.775048    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="1d9bc028-587d-4213-a322-9ea0364a7109" containerName="console"
May 20 15:35:04.778002 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.777977    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:04.781337 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.781320    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 15:35:04.787723 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.787700    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qs4db"]
May 20 15:35:04.835451 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.835422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcacd4aa-c77b-4759-95d6-223503059be9-dbus\") pod \"global-pull-secret-syncer-qs4db\" (UID: \"bcacd4aa-c77b-4759-95d6-223503059be9\") " pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:04.835622 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.835467    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcacd4aa-c77b-4759-95d6-223503059be9-original-pull-secret\") pod \"global-pull-secret-syncer-qs4db\" (UID: \"bcacd4aa-c77b-4759-95d6-223503059be9\") " pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:04.835622 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.835504    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcacd4aa-c77b-4759-95d6-223503059be9-kubelet-config\") pod \"global-pull-secret-syncer-qs4db\" (UID: \"bcacd4aa-c77b-4759-95d6-223503059be9\") " pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:04.936292 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.936257    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcacd4aa-c77b-4759-95d6-223503059be9-original-pull-secret\") pod \"global-pull-secret-syncer-qs4db\" (UID: \"bcacd4aa-c77b-4759-95d6-223503059be9\") " pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:04.936549 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.936324    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcacd4aa-c77b-4759-95d6-223503059be9-kubelet-config\") pod \"global-pull-secret-syncer-qs4db\" (UID: \"bcacd4aa-c77b-4759-95d6-223503059be9\") " pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:04.936549 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.936429    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcacd4aa-c77b-4759-95d6-223503059be9-dbus\") pod \"global-pull-secret-syncer-qs4db\" (UID: \"bcacd4aa-c77b-4759-95d6-223503059be9\") " pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:04.936549 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.936442    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcacd4aa-c77b-4759-95d6-223503059be9-kubelet-config\") pod \"global-pull-secret-syncer-qs4db\" (UID: \"bcacd4aa-c77b-4759-95d6-223503059be9\") " pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:04.936688 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.936578    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcacd4aa-c77b-4759-95d6-223503059be9-dbus\") pod \"global-pull-secret-syncer-qs4db\" (UID: \"bcacd4aa-c77b-4759-95d6-223503059be9\") " pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:04.938491 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:04.938470    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcacd4aa-c77b-4759-95d6-223503059be9-original-pull-secret\") pod \"global-pull-secret-syncer-qs4db\" (UID: \"bcacd4aa-c77b-4759-95d6-223503059be9\") " pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:05.087143 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:05.087111    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qs4db"
May 20 15:35:05.217969 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:05.217848    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qs4db"]
May 20 15:35:05.220595 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:35:05.220564    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcacd4aa_c77b_4759_95d6_223503059be9.slice/crio-8606c174013f75fa7fb1dce465216b81a46bd93737a2dff653fd3218199b84f3 WatchSource:0}: Error finding container 8606c174013f75fa7fb1dce465216b81a46bd93737a2dff653fd3218199b84f3: Status 404 returned error can't find the container with id 8606c174013f75fa7fb1dce465216b81a46bd93737a2dff653fd3218199b84f3
May 20 15:35:05.585224 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:05.585188    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qs4db" event={"ID":"bcacd4aa-c77b-4759-95d6-223503059be9","Type":"ContainerStarted","Data":"8606c174013f75fa7fb1dce465216b81a46bd93737a2dff653fd3218199b84f3"}
May 20 15:35:07.590714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:07.590664    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5ff5947f47-xbm4d" podUID="e4958e27-4626-4556-b172-5ae410bd0daa" containerName="console" containerID="cri-o://801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed" gracePeriod=15
May 20 15:35:08.927510 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:08.927481    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ff5947f47-xbm4d_e4958e27-4626-4556-b172-5ae410bd0daa/console/0.log"
May 20 15:35:08.927934 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:08.927562    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:35:09.093880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.093840    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-trusted-ca-bundle\") pod \"e4958e27-4626-4556-b172-5ae410bd0daa\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") "
May 20 15:35:09.094089 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.093963    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-serving-cert\") pod \"e4958e27-4626-4556-b172-5ae410bd0daa\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") "
May 20 15:35:09.094089 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.093990    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-console-config\") pod \"e4958e27-4626-4556-b172-5ae410bd0daa\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") "
May 20 15:35:09.094089 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.094012    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-oauth-serving-cert\") pod \"e4958e27-4626-4556-b172-5ae410bd0daa\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") "
May 20 15:35:09.094089 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.094038    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-oauth-config\") pod \"e4958e27-4626-4556-b172-5ae410bd0daa\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") "
May 20 15:35:09.094089 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.094065    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-service-ca\") pod \"e4958e27-4626-4556-b172-5ae410bd0daa\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") "
May 20 15:35:09.094336 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.094128    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tldqr\" (UniqueName: \"kubernetes.io/projected/e4958e27-4626-4556-b172-5ae410bd0daa-kube-api-access-tldqr\") pod \"e4958e27-4626-4556-b172-5ae410bd0daa\" (UID: \"e4958e27-4626-4556-b172-5ae410bd0daa\") "
May 20 15:35:09.094468 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.094437    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e4958e27-4626-4556-b172-5ae410bd0daa" (UID: "e4958e27-4626-4556-b172-5ae410bd0daa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:09.094584 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.094447    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-console-config" (OuterVolumeSpecName: "console-config") pod "e4958e27-4626-4556-b172-5ae410bd0daa" (UID: "e4958e27-4626-4556-b172-5ae410bd0daa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:09.094677 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.094645    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e4958e27-4626-4556-b172-5ae410bd0daa" (UID: "e4958e27-4626-4556-b172-5ae410bd0daa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:09.094930 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.094890    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-service-ca" (OuterVolumeSpecName: "service-ca") pod "e4958e27-4626-4556-b172-5ae410bd0daa" (UID: "e4958e27-4626-4556-b172-5ae410bd0daa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:09.096669 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.096638    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e4958e27-4626-4556-b172-5ae410bd0daa" (UID: "e4958e27-4626-4556-b172-5ae410bd0daa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:35:09.097016 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.096990    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4958e27-4626-4556-b172-5ae410bd0daa-kube-api-access-tldqr" (OuterVolumeSpecName: "kube-api-access-tldqr") pod "e4958e27-4626-4556-b172-5ae410bd0daa" (UID: "e4958e27-4626-4556-b172-5ae410bd0daa"). InnerVolumeSpecName "kube-api-access-tldqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:35:09.097016 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.096994    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e4958e27-4626-4556-b172-5ae410bd0daa" (UID: "e4958e27-4626-4556-b172-5ae410bd0daa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:35:09.195591 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.195547    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tldqr\" (UniqueName: \"kubernetes.io/projected/e4958e27-4626-4556-b172-5ae410bd0daa-kube-api-access-tldqr\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:35:09.195591 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.195589    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:35:09.195800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.195605    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:35:09.195800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.195620    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:35:09.195800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.195636    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:35:09.195800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.195649    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4958e27-4626-4556-b172-5ae410bd0daa-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:35:09.195800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.195661    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4958e27-4626-4556-b172-5ae410bd0daa-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:35:09.600276 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.600248    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ff5947f47-xbm4d_e4958e27-4626-4556-b172-5ae410bd0daa/console/0.log"
May 20 15:35:09.600481 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.600288    2570 generic.go:358] "Generic (PLEG): container finished" podID="e4958e27-4626-4556-b172-5ae410bd0daa" containerID="801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed" exitCode=2
May 20 15:35:09.600481 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.600399    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff5947f47-xbm4d" event={"ID":"e4958e27-4626-4556-b172-5ae410bd0daa","Type":"ContainerDied","Data":"801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed"}
May 20 15:35:09.600481 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.600412    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff5947f47-xbm4d"
May 20 15:35:09.600481 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.600437    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff5947f47-xbm4d" event={"ID":"e4958e27-4626-4556-b172-5ae410bd0daa","Type":"ContainerDied","Data":"59a55589f4ed298a80f241bc389abf13c6b75ef33e2a2a2a038f49832cecbd4e"}
May 20 15:35:09.600481 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.600453    2570 scope.go:117] "RemoveContainer" containerID="801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed"
May 20 15:35:09.601957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.601931    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qs4db" event={"ID":"bcacd4aa-c77b-4759-95d6-223503059be9","Type":"ContainerStarted","Data":"700de49092418f3b28f5c4d4fd9e8f884cd3dc60cffbc989171bf56a377d3277"}
May 20 15:35:09.609294 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.609273    2570 scope.go:117] "RemoveContainer" containerID="801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed"
May 20 15:35:09.609615 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:35:09.609591    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed\": container with ID starting with 801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed not found: ID does not exist" containerID="801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed"
May 20 15:35:09.609695 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.609626    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed"} err="failed to get container status \"801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed\": rpc error: code = NotFound desc = could not find container \"801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed\": container with ID starting with 801a4311fdb6570909897122740dd84b9ca9912698c62915739d3c86b21327ed not found: ID does not exist"
May 20 15:35:09.622032 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.621977    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qs4db" podStartSLOduration=2.045370455 podStartE2EDuration="5.621963272s" podCreationTimestamp="2026-05-20 15:35:04 +0000 UTC" firstStartedPulling="2026-05-20 15:35:05.222183936 +0000 UTC m=+194.052279835" lastFinishedPulling="2026-05-20 15:35:08.798776736 +0000 UTC m=+197.628872652" observedRunningTime="2026-05-20 15:35:09.620568109 +0000 UTC m=+198.450664039" watchObservedRunningTime="2026-05-20 15:35:09.621963272 +0000 UTC m=+198.452059192"
May 20 15:35:09.635221 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.635179    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff5947f47-xbm4d"]
May 20 15:35:09.639149 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.639120    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ff5947f47-xbm4d"]
May 20 15:35:09.760682 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:35:09.760641    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4958e27-4626-4556-b172-5ae410bd0daa" path="/var/lib/kubelet/pods/e4958e27-4626-4556-b172-5ae410bd0daa/volumes"
May 20 15:36:51.638261 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:36:51.638231    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:36:51.638765 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:36:51.638272    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:36:51.641751 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:36:51.641732    2570 kubelet.go:1628] "Image garbage collection succeeded"
May 20 15:38:35.567688 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.567651    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-xdjm4"]
May 20 15:38:35.568085 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.567987    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4958e27-4626-4556-b172-5ae410bd0daa" containerName="console"
May 20 15:38:35.568085 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.567999    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4958e27-4626-4556-b172-5ae410bd0daa" containerName="console"
May 20 15:38:35.568085 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.568067    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="e4958e27-4626-4556-b172-5ae410bd0daa" containerName="console"
May 20 15:38:35.571023 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.571005    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:35.573909 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.573888    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-qbltd\""
May 20 15:38:35.574053 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.573946    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 15:38:35.574711 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.574696    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 15:38:35.574834 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.574783    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 20 15:38:35.583546 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.583524    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-xdjm4"]
May 20 15:38:35.586090 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.586073    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-shczj"]
May 20 15:38:35.589282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.589264    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:35.591889 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.591862    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 20 15:38:35.591983 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.591944    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-bzjg2\""
May 20 15:38:35.598964 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.598933    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-shczj"]
May 20 15:38:35.609411 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.609384    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhg4h\" (UniqueName: \"kubernetes.io/projected/76a36312-7a13-48b2-919e-4ccc398882a0-kube-api-access-bhg4h\") pod \"model-serving-api-86f7b4b499-xdjm4\" (UID: \"76a36312-7a13-48b2-919e-4ccc398882a0\") " pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:35.609526 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.609489    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a36312-7a13-48b2-919e-4ccc398882a0-tls-certs\") pod \"model-serving-api-86f7b4b499-xdjm4\" (UID: \"76a36312-7a13-48b2-919e-4ccc398882a0\") " pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:35.710900 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.710860    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a36312-7a13-48b2-919e-4ccc398882a0-tls-certs\") pod \"model-serving-api-86f7b4b499-xdjm4\" (UID: \"76a36312-7a13-48b2-919e-4ccc398882a0\") " pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:35.711102 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.710926    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdjr\" (UniqueName: \"kubernetes.io/projected/9b9981b2-3a71-401f-b54b-36bea294fd25-kube-api-access-qqdjr\") pod \"odh-model-controller-696fc77849-shczj\" (UID: \"9b9981b2-3a71-401f-b54b-36bea294fd25\") " pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:35.711102 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.710970    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bhg4h\" (UniqueName: \"kubernetes.io/projected/76a36312-7a13-48b2-919e-4ccc398882a0-kube-api-access-bhg4h\") pod \"model-serving-api-86f7b4b499-xdjm4\" (UID: \"76a36312-7a13-48b2-919e-4ccc398882a0\") " pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:35.711102 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.711009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b9981b2-3a71-401f-b54b-36bea294fd25-cert\") pod \"odh-model-controller-696fc77849-shczj\" (UID: \"9b9981b2-3a71-401f-b54b-36bea294fd25\") " pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:35.711102 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:38:35.711022    2570 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
May 20 15:38:35.711102 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:38:35.711081    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76a36312-7a13-48b2-919e-4ccc398882a0-tls-certs podName:76a36312-7a13-48b2-919e-4ccc398882a0 nodeName:}" failed. No retries permitted until 2026-05-20 15:38:36.211060665 +0000 UTC m=+405.041156569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/76a36312-7a13-48b2-919e-4ccc398882a0-tls-certs") pod "model-serving-api-86f7b4b499-xdjm4" (UID: "76a36312-7a13-48b2-919e-4ccc398882a0") : secret "model-serving-api-tls" not found
May 20 15:38:35.720719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.720690    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhg4h\" (UniqueName: \"kubernetes.io/projected/76a36312-7a13-48b2-919e-4ccc398882a0-kube-api-access-bhg4h\") pod \"model-serving-api-86f7b4b499-xdjm4\" (UID: \"76a36312-7a13-48b2-919e-4ccc398882a0\") " pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:35.811906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.811863    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdjr\" (UniqueName: \"kubernetes.io/projected/9b9981b2-3a71-401f-b54b-36bea294fd25-kube-api-access-qqdjr\") pod \"odh-model-controller-696fc77849-shczj\" (UID: \"9b9981b2-3a71-401f-b54b-36bea294fd25\") " pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:35.812104 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.811942    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b9981b2-3a71-401f-b54b-36bea294fd25-cert\") pod \"odh-model-controller-696fc77849-shczj\" (UID: \"9b9981b2-3a71-401f-b54b-36bea294fd25\") " pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:35.814415 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.814396    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b9981b2-3a71-401f-b54b-36bea294fd25-cert\") pod \"odh-model-controller-696fc77849-shczj\" (UID: \"9b9981b2-3a71-401f-b54b-36bea294fd25\") " pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:35.824078 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.824014    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdjr\" (UniqueName: \"kubernetes.io/projected/9b9981b2-3a71-401f-b54b-36bea294fd25-kube-api-access-qqdjr\") pod \"odh-model-controller-696fc77849-shczj\" (UID: \"9b9981b2-3a71-401f-b54b-36bea294fd25\") " pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:35.901203 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:35.901169    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:36.027404 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:36.027354    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-shczj"]
May 20 15:38:36.030414 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:38:36.030388    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9981b2_3a71_401f_b54b_36bea294fd25.slice/crio-fbc3b28a7bf24c936505815fd13f9f61c9b2782188912c62f4569bbda6a1bfc3 WatchSource:0}: Error finding container fbc3b28a7bf24c936505815fd13f9f61c9b2782188912c62f4569bbda6a1bfc3: Status 404 returned error can't find the container with id fbc3b28a7bf24c936505815fd13f9f61c9b2782188912c62f4569bbda6a1bfc3
May 20 15:38:36.031596 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:36.031581    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:38:36.215220 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:36.215137    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a36312-7a13-48b2-919e-4ccc398882a0-tls-certs\") pod \"model-serving-api-86f7b4b499-xdjm4\" (UID: \"76a36312-7a13-48b2-919e-4ccc398882a0\") " pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:36.217518 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:36.217490    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a36312-7a13-48b2-919e-4ccc398882a0-tls-certs\") pod \"model-serving-api-86f7b4b499-xdjm4\" (UID: \"76a36312-7a13-48b2-919e-4ccc398882a0\") " pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:36.225490 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:36.225464    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-shczj" event={"ID":"9b9981b2-3a71-401f-b54b-36bea294fd25","Type":"ContainerStarted","Data":"fbc3b28a7bf24c936505815fd13f9f61c9b2782188912c62f4569bbda6a1bfc3"}
May 20 15:38:36.482502 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:36.482418    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:36.606241 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:36.606207    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-xdjm4"]
May 20 15:38:36.610572 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:38:36.610541    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a36312_7a13_48b2_919e_4ccc398882a0.slice/crio-a1a4849cae318f6e69d994fc372f9d5c6ce992a0a42d28c7b262c84c78dedfc8 WatchSource:0}: Error finding container a1a4849cae318f6e69d994fc372f9d5c6ce992a0a42d28c7b262c84c78dedfc8: Status 404 returned error can't find the container with id a1a4849cae318f6e69d994fc372f9d5c6ce992a0a42d28c7b262c84c78dedfc8
May 20 15:38:37.230239 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:37.230190    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-xdjm4" event={"ID":"76a36312-7a13-48b2-919e-4ccc398882a0","Type":"ContainerStarted","Data":"a1a4849cae318f6e69d994fc372f9d5c6ce992a0a42d28c7b262c84c78dedfc8"}
May 20 15:38:40.241950 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:40.241834    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-shczj" event={"ID":"9b9981b2-3a71-401f-b54b-36bea294fd25","Type":"ContainerStarted","Data":"675e61a143564e59db596c562fdc3fae6dadde0a486130772316d483496b15d8"}
May 20 15:38:40.241950 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:40.241944    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:40.243467 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:40.243441    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-xdjm4" event={"ID":"76a36312-7a13-48b2-919e-4ccc398882a0","Type":"ContainerStarted","Data":"2fe018177eca14687e72ff05c395d907513d9209019763acb8a173cc2f3278c1"}
May 20 15:38:40.243630 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:40.243587    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:40.265654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:40.265589    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-shczj" podStartSLOduration=1.336048741 podStartE2EDuration="5.2655705s" podCreationTimestamp="2026-05-20 15:38:35 +0000 UTC" firstStartedPulling="2026-05-20 15:38:36.031699637 +0000 UTC m=+404.861795536" lastFinishedPulling="2026-05-20 15:38:39.961221383 +0000 UTC m=+408.791317295" observedRunningTime="2026-05-20 15:38:40.263561283 +0000 UTC m=+409.093657205" watchObservedRunningTime="2026-05-20 15:38:40.2655705 +0000 UTC m=+409.095666420"
May 20 15:38:40.288384 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:40.288307    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-xdjm4" podStartSLOduration=1.887001785 podStartE2EDuration="5.28829029s" podCreationTimestamp="2026-05-20 15:38:35 +0000 UTC" firstStartedPulling="2026-05-20 15:38:36.612902394 +0000 UTC m=+405.442998294" lastFinishedPulling="2026-05-20 15:38:40.0141909 +0000 UTC m=+408.844286799" observedRunningTime="2026-05-20 15:38:40.286796751 +0000 UTC m=+409.116892674" watchObservedRunningTime="2026-05-20 15:38:40.28829029 +0000 UTC m=+409.118386283"
May 20 15:38:51.248660 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:51.248629    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-shczj"
May 20 15:38:51.250559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:51.250537    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-xdjm4"
May 20 15:38:52.144351 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.144312    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-fbbbs"]
May 20 15:38:52.148005 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.147984    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fbbbs"
May 20 15:38:52.150900 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.150657    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xzlwk\""
May 20 15:38:52.151469 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.151444    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 20 15:38:52.152805 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.152781    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-fbbbs"]
May 20 15:38:52.277155 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.277117    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvnv\" (UniqueName: \"kubernetes.io/projected/0f9358a9-7df4-4a83-b1ac-420fbaa4231f-kube-api-access-lwvnv\") pod \"s3-init-fbbbs\" (UID: \"0f9358a9-7df4-4a83-b1ac-420fbaa4231f\") " pod="kserve/s3-init-fbbbs"
May 20 15:38:52.377957 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.377921    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvnv\" (UniqueName: \"kubernetes.io/projected/0f9358a9-7df4-4a83-b1ac-420fbaa4231f-kube-api-access-lwvnv\") pod \"s3-init-fbbbs\" (UID: \"0f9358a9-7df4-4a83-b1ac-420fbaa4231f\") " pod="kserve/s3-init-fbbbs"
May 20 15:38:52.386114 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.386090    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvnv\" (UniqueName: \"kubernetes.io/projected/0f9358a9-7df4-4a83-b1ac-420fbaa4231f-kube-api-access-lwvnv\") pod \"s3-init-fbbbs\" (UID: \"0f9358a9-7df4-4a83-b1ac-420fbaa4231f\") " pod="kserve/s3-init-fbbbs"
May 20 15:38:52.468777 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.468680    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fbbbs"
May 20 15:38:52.605740 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:52.605567    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-fbbbs"]
May 20 15:38:52.608573 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:38:52.608543    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f9358a9_7df4_4a83_b1ac_420fbaa4231f.slice/crio-2490a1dd37646330a84d52874911e54019a49229ebbdb85da6ee58185faea412 WatchSource:0}: Error finding container 2490a1dd37646330a84d52874911e54019a49229ebbdb85da6ee58185faea412: Status 404 returned error can't find the container with id 2490a1dd37646330a84d52874911e54019a49229ebbdb85da6ee58185faea412
May 20 15:38:53.285779 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:53.285739    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fbbbs" event={"ID":"0f9358a9-7df4-4a83-b1ac-420fbaa4231f","Type":"ContainerStarted","Data":"2490a1dd37646330a84d52874911e54019a49229ebbdb85da6ee58185faea412"}
May 20 15:38:58.303852 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:58.303812    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fbbbs" event={"ID":"0f9358a9-7df4-4a83-b1ac-420fbaa4231f","Type":"ContainerStarted","Data":"761684cff38b6d523f95aa58be31d1cb0cb01806c4d254a0295b8b1ed7ee4f4d"}
May 20 15:38:58.320811 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:38:58.320738    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-fbbbs" podStartSLOduration=1.5483850810000002 podStartE2EDuration="6.320722462s" podCreationTimestamp="2026-05-20 15:38:52 +0000 UTC" firstStartedPulling="2026-05-20 15:38:52.610799146 +0000 UTC m=+421.440895045" lastFinishedPulling="2026-05-20 15:38:57.383136522 +0000 UTC m=+426.213232426" observedRunningTime="2026-05-20 15:38:58.31826874 +0000 UTC m=+427.148364661" watchObservedRunningTime="2026-05-20 15:38:58.320722462 +0000 UTC m=+427.150818383"
May 20 15:39:01.314844 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:01.314811    2570 generic.go:358] "Generic (PLEG): container finished" podID="0f9358a9-7df4-4a83-b1ac-420fbaa4231f" containerID="761684cff38b6d523f95aa58be31d1cb0cb01806c4d254a0295b8b1ed7ee4f4d" exitCode=0
May 20 15:39:01.315256 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:01.314883    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fbbbs" event={"ID":"0f9358a9-7df4-4a83-b1ac-420fbaa4231f","Type":"ContainerDied","Data":"761684cff38b6d523f95aa58be31d1cb0cb01806c4d254a0295b8b1ed7ee4f4d"}
May 20 15:39:02.446678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:02.446651    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fbbbs"
May 20 15:39:02.471173 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:02.471145    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwvnv\" (UniqueName: \"kubernetes.io/projected/0f9358a9-7df4-4a83-b1ac-420fbaa4231f-kube-api-access-lwvnv\") pod \"0f9358a9-7df4-4a83-b1ac-420fbaa4231f\" (UID: \"0f9358a9-7df4-4a83-b1ac-420fbaa4231f\") "
May 20 15:39:02.473321 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:02.473284    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9358a9-7df4-4a83-b1ac-420fbaa4231f-kube-api-access-lwvnv" (OuterVolumeSpecName: "kube-api-access-lwvnv") pod "0f9358a9-7df4-4a83-b1ac-420fbaa4231f" (UID: "0f9358a9-7df4-4a83-b1ac-420fbaa4231f"). InnerVolumeSpecName "kube-api-access-lwvnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:39:02.572442 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:02.572340    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvnv\" (UniqueName: \"kubernetes.io/projected/0f9358a9-7df4-4a83-b1ac-420fbaa4231f-kube-api-access-lwvnv\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:39:03.322982 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:03.322946    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fbbbs" event={"ID":"0f9358a9-7df4-4a83-b1ac-420fbaa4231f","Type":"ContainerDied","Data":"2490a1dd37646330a84d52874911e54019a49229ebbdb85da6ee58185faea412"}
May 20 15:39:03.322982 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:03.322980    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2490a1dd37646330a84d52874911e54019a49229ebbdb85da6ee58185faea412"
May 20 15:39:03.322982 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:03.322987    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fbbbs"
May 20 15:39:14.683192 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.683112    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"]
May 20 15:39:14.683559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.683499    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9358a9-7df4-4a83-b1ac-420fbaa4231f" containerName="s3-init"
May 20 15:39:14.683559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.683512    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9358a9-7df4-4a83-b1ac-420fbaa4231f" containerName="s3-init"
May 20 15:39:14.683639 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.683601    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="0f9358a9-7df4-4a83-b1ac-420fbaa4231f" containerName="s3-init"
May 20 15:39:14.689379 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.689348    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.692049 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.692020    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qxvlr\""
May 20 15:39:14.692049 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.692047    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:39:14.692238 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.692068    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-be7d1-predictor-serving-cert\""
May 20 15:39:14.692238 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.692068    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:39:14.692238 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.692068    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\""
May 20 15:39:14.700523 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.700501    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"]
May 20 15:39:14.781162 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.781125    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28616225-17ab-40a2-8c93-3f711b09517d-isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.781316 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.781184    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28616225-17ab-40a2-8c93-3f711b09517d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.781316 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.781278    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gff49\" (UniqueName: \"kubernetes.io/projected/28616225-17ab-40a2-8c93-3f711b09517d-kube-api-access-gff49\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.781422 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.781324    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28616225-17ab-40a2-8c93-3f711b09517d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.882566 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.882514    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gff49\" (UniqueName: \"kubernetes.io/projected/28616225-17ab-40a2-8c93-3f711b09517d-kube-api-access-gff49\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.882743 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.882591    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28616225-17ab-40a2-8c93-3f711b09517d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.882743 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.882631    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28616225-17ab-40a2-8c93-3f711b09517d-isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.882743 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.882696    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28616225-17ab-40a2-8c93-3f711b09517d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.882908 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:39:14.882836    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-be7d1-predictor-serving-cert" not found
May 20 15:39:14.882908 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:39:14.882897    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28616225-17ab-40a2-8c93-3f711b09517d-proxy-tls podName:28616225-17ab-40a2-8c93-3f711b09517d nodeName:}" failed. No retries permitted until 2026-05-20 15:39:15.382875242 +0000 UTC m=+444.212971144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/28616225-17ab-40a2-8c93-3f711b09517d-proxy-tls") pod "isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" (UID: "28616225-17ab-40a2-8c93-3f711b09517d") : secret "isvc-raw-sklearn-batcher-be7d1-predictor-serving-cert" not found
May 20 15:39:14.883079 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.883052    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28616225-17ab-40a2-8c93-3f711b09517d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.883318 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.883297    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28616225-17ab-40a2-8c93-3f711b09517d-isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:14.891472 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:14.891443    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gff49\" (UniqueName: \"kubernetes.io/projected/28616225-17ab-40a2-8c93-3f711b09517d-kube-api-access-gff49\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:15.387280 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:15.387252    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28616225-17ab-40a2-8c93-3f711b09517d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:15.389651 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:15.389624    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28616225-17ab-40a2-8c93-3f711b09517d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:15.599311 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:15.599275    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:15.733832 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:15.733811    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"]
May 20 15:39:15.736659 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:39:15.736627    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28616225_17ab_40a2_8c93_3f711b09517d.slice/crio-f256893216aa8769e56e4af2228fc05155a7cd6ddae69b5f2661d8412d8d997f WatchSource:0}: Error finding container f256893216aa8769e56e4af2228fc05155a7cd6ddae69b5f2661d8412d8d997f: Status 404 returned error can't find the container with id f256893216aa8769e56e4af2228fc05155a7cd6ddae69b5f2661d8412d8d997f
May 20 15:39:16.365453 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:16.365406    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerStarted","Data":"f256893216aa8769e56e4af2228fc05155a7cd6ddae69b5f2661d8412d8d997f"}
May 20 15:39:20.381307 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:20.381268    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerStarted","Data":"8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5"}
May 20 15:39:24.394387 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:24.394337    2570 generic.go:358] "Generic (PLEG): container finished" podID="28616225-17ab-40a2-8c93-3f711b09517d" containerID="8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5" exitCode=0
May 20 15:39:24.394735 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:24.394394    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerDied","Data":"8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5"}
May 20 15:39:26.842473 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.841928    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-697d8c7bb4-wbrl2"]
May 20 15:39:26.846201 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.845852    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:26.858927 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.857829    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-697d8c7bb4-wbrl2"]
May 20 15:39:26.995619 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.995584    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-service-ca\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:26.995800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.995639    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgsb5\" (UniqueName: \"kubernetes.io/projected/84f757ad-1705-4d2c-a2a8-8f419de18ddf-kube-api-access-tgsb5\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:26.995800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.995676    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-trusted-ca-bundle\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:26.995800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.995754    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84f757ad-1705-4d2c-a2a8-8f419de18ddf-console-serving-cert\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:26.995800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.995789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-console-config\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:26.996015 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.995851    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-oauth-serving-cert\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:26.996015 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:26.995934    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84f757ad-1705-4d2c-a2a8-8f419de18ddf-console-oauth-config\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.097798 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.097308    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84f757ad-1705-4d2c-a2a8-8f419de18ddf-console-oauth-config\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.097798 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.097403    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-service-ca\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.097798 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.097444    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsb5\" (UniqueName: \"kubernetes.io/projected/84f757ad-1705-4d2c-a2a8-8f419de18ddf-kube-api-access-tgsb5\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.097798 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.097470    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-trusted-ca-bundle\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.097798 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.097550    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84f757ad-1705-4d2c-a2a8-8f419de18ddf-console-serving-cert\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.097798 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.097609    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-console-config\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.097798 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.097652    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-oauth-serving-cert\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.098469 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.098325    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-service-ca\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.098469 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.098435    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-oauth-serving-cert\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.098768 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.098707    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-console-config\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.098900 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.098847    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84f757ad-1705-4d2c-a2a8-8f419de18ddf-trusted-ca-bundle\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.101015 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.100972    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84f757ad-1705-4d2c-a2a8-8f419de18ddf-console-serving-cert\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.101015 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.101004    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84f757ad-1705-4d2c-a2a8-8f419de18ddf-console-oauth-config\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.106690 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.106654    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsb5\" (UniqueName: \"kubernetes.io/projected/84f757ad-1705-4d2c-a2a8-8f419de18ddf-kube-api-access-tgsb5\") pod \"console-697d8c7bb4-wbrl2\" (UID: \"84f757ad-1705-4d2c-a2a8-8f419de18ddf\") " pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.161897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.161827    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:27.318797 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.318719    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-697d8c7bb4-wbrl2"]
May 20 15:39:27.322807 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:39:27.322772    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f757ad_1705_4d2c_a2a8_8f419de18ddf.slice/crio-86be8ab1586d1b30f71a941ef6880b4d8825987d2ed364e4d1f07445c0bbe50f WatchSource:0}: Error finding container 86be8ab1586d1b30f71a941ef6880b4d8825987d2ed364e4d1f07445c0bbe50f: Status 404 returned error can't find the container with id 86be8ab1586d1b30f71a941ef6880b4d8825987d2ed364e4d1f07445c0bbe50f
May 20 15:39:27.408928 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:27.408894    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697d8c7bb4-wbrl2" event={"ID":"84f757ad-1705-4d2c-a2a8-8f419de18ddf","Type":"ContainerStarted","Data":"86be8ab1586d1b30f71a941ef6880b4d8825987d2ed364e4d1f07445c0bbe50f"}
May 20 15:39:28.415481 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:28.415416    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697d8c7bb4-wbrl2" event={"ID":"84f757ad-1705-4d2c-a2a8-8f419de18ddf","Type":"ContainerStarted","Data":"a961f0a0689888d179d74e971dbea1c90bf36fdeb8b5e70dc49929d6f1d1238c"}
May 20 15:39:28.435954 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:28.435886    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-697d8c7bb4-wbrl2" podStartSLOduration=2.435864423 podStartE2EDuration="2.435864423s" podCreationTimestamp="2026-05-20 15:39:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:39:28.432468414 +0000 UTC m=+457.262564336" watchObservedRunningTime="2026-05-20 15:39:28.435864423 +0000 UTC m=+457.265960345"
May 20 15:39:37.162465 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:37.162436    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:37.162763 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:37.162484    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:37.167454 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:37.167427    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:37.456461 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:37.456360    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerStarted","Data":"82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637"}
May 20 15:39:37.460612 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:37.460588    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-697d8c7bb4-wbrl2"
May 20 15:39:37.510024 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:37.509989    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf6459878-zhk9m"]
May 20 15:39:40.471727 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:40.471690    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerStarted","Data":"4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402"}
May 20 15:39:42.481457 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:42.481419    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerStarted","Data":"6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f"}
May 20 15:39:42.481886 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:42.481687    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:42.481886 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:42.481839    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:42.483305 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:42.483272    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:39:42.503405 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:42.503317    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podStartSLOduration=2.248778041 podStartE2EDuration="28.503297064s" podCreationTimestamp="2026-05-20 15:39:14 +0000 UTC" firstStartedPulling="2026-05-20 15:39:15.738614901 +0000 UTC m=+444.568710801" lastFinishedPulling="2026-05-20 15:39:41.993133913 +0000 UTC m=+470.823229824" observedRunningTime="2026-05-20 15:39:42.501278592 +0000 UTC m=+471.331374540" watchObservedRunningTime="2026-05-20 15:39:42.503297064 +0000 UTC m=+471.333392985"
May 20 15:39:43.484762 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:43.484727    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:43.485301 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:43.484898    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:39:43.485812 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:43.485789    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:39:44.488725 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:44.488675    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:39:44.489165 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:44.489125    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:39:44.492468 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:44.492447    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:39:45.492198 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:45.492151    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:39:45.492658 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:45.492511    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:39:55.492292 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:55.492245    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:39:55.492781 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:39:55.492706    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:40:02.530895 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.530839    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5bf6459878-zhk9m" podUID="10d66096-0b67-4cbd-be29-2c08f88d145e" containerName="console" containerID="cri-o://467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1" gracePeriod=15
May 20 15:40:02.790683 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.790662    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bf6459878-zhk9m_10d66096-0b67-4cbd-be29-2c08f88d145e/console/0.log"
May 20 15:40:02.790807 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.790723    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:40:02.830559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.830527    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-trusted-ca-bundle\") pod \"10d66096-0b67-4cbd-be29-2c08f88d145e\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") "
May 20 15:40:02.830681 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.830585    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-console-config\") pod \"10d66096-0b67-4cbd-be29-2c08f88d145e\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") "
May 20 15:40:02.830681 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.830629    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-serving-cert\") pod \"10d66096-0b67-4cbd-be29-2c08f88d145e\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") "
May 20 15:40:02.830681 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.830657    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-oauth-serving-cert\") pod \"10d66096-0b67-4cbd-be29-2c08f88d145e\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") "
May 20 15:40:02.830826 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.830682    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpqpn\" (UniqueName: \"kubernetes.io/projected/10d66096-0b67-4cbd-be29-2c08f88d145e-kube-api-access-lpqpn\") pod \"10d66096-0b67-4cbd-be29-2c08f88d145e\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") "
May 20 15:40:02.830826 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.830705    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-oauth-config\") pod \"10d66096-0b67-4cbd-be29-2c08f88d145e\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") "
May 20 15:40:02.830826 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.830746    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-service-ca\") pod \"10d66096-0b67-4cbd-be29-2c08f88d145e\" (UID: \"10d66096-0b67-4cbd-be29-2c08f88d145e\") "
May 20 15:40:02.831114 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.831079    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-console-config" (OuterVolumeSpecName: "console-config") pod "10d66096-0b67-4cbd-be29-2c08f88d145e" (UID: "10d66096-0b67-4cbd-be29-2c08f88d145e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:40:02.831245 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.831114    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "10d66096-0b67-4cbd-be29-2c08f88d145e" (UID: "10d66096-0b67-4cbd-be29-2c08f88d145e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:40:02.831313 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.831281    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-service-ca" (OuterVolumeSpecName: "service-ca") pod "10d66096-0b67-4cbd-be29-2c08f88d145e" (UID: "10d66096-0b67-4cbd-be29-2c08f88d145e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:40:02.831391 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.831338    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "10d66096-0b67-4cbd-be29-2c08f88d145e" (UID: "10d66096-0b67-4cbd-be29-2c08f88d145e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:40:02.833121 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.833098    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "10d66096-0b67-4cbd-be29-2c08f88d145e" (UID: "10d66096-0b67-4cbd-be29-2c08f88d145e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:40:02.833218 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.833147    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d66096-0b67-4cbd-be29-2c08f88d145e-kube-api-access-lpqpn" (OuterVolumeSpecName: "kube-api-access-lpqpn") pod "10d66096-0b67-4cbd-be29-2c08f88d145e" (UID: "10d66096-0b67-4cbd-be29-2c08f88d145e"). InnerVolumeSpecName "kube-api-access-lpqpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:40:02.833427 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.833404    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "10d66096-0b67-4cbd-be29-2c08f88d145e" (UID: "10d66096-0b67-4cbd-be29-2c08f88d145e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:40:02.931978 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.931938    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:40:02.931978 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.931973    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:40:02.931978 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.931984    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:40:02.932209 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.931992    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:40:02.932209 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.932001    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10d66096-0b67-4cbd-be29-2c08f88d145e-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:40:02.932209 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.932010    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpqpn\" (UniqueName: \"kubernetes.io/projected/10d66096-0b67-4cbd-be29-2c08f88d145e-kube-api-access-lpqpn\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:40:02.932209 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:02.932020    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10d66096-0b67-4cbd-be29-2c08f88d145e-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:40:03.555106 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.555081    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bf6459878-zhk9m_10d66096-0b67-4cbd-be29-2c08f88d145e/console/0.log"
May 20 15:40:03.555508 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.555119    2570 generic.go:358] "Generic (PLEG): container finished" podID="10d66096-0b67-4cbd-be29-2c08f88d145e" containerID="467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1" exitCode=2
May 20 15:40:03.555508 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.555153    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf6459878-zhk9m" event={"ID":"10d66096-0b67-4cbd-be29-2c08f88d145e","Type":"ContainerDied","Data":"467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1"}
May 20 15:40:03.555508 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.555196    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf6459878-zhk9m"
May 20 15:40:03.555508 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.555210    2570 scope.go:117] "RemoveContainer" containerID="467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1"
May 20 15:40:03.555508 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.555198    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf6459878-zhk9m" event={"ID":"10d66096-0b67-4cbd-be29-2c08f88d145e","Type":"ContainerDied","Data":"5887dae25918b8b97a7e07f76520eeeeb2f4723806d84677bce809d23c06b3fd"}
May 20 15:40:03.563966 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.563949    2570 scope.go:117] "RemoveContainer" containerID="467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1"
May 20 15:40:03.564206 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:40:03.564188    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1\": container with ID starting with 467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1 not found: ID does not exist" containerID="467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1"
May 20 15:40:03.564257 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.564214    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1"} err="failed to get container status \"467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1\": rpc error: code = NotFound desc = could not find container \"467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1\": container with ID starting with 467a5b54b8b764a4c8eb0fc457f92c8ffc730004e117950242916be59afe6da1 not found: ID does not exist"
May 20 15:40:03.576655 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.576628    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf6459878-zhk9m"]
May 20 15:40:03.580994 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.580970    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bf6459878-zhk9m"]
May 20 15:40:03.759604 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:03.759573    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10d66096-0b67-4cbd-be29-2c08f88d145e" path="/var/lib/kubelet/pods/10d66096-0b67-4cbd-be29-2c08f88d145e/volumes"
May 20 15:40:05.492535 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:05.492488    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:40:05.492981 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:05.492956    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:40:15.492422 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:15.492353    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:40:15.492948 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:15.492887    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:40:25.492479 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:25.492428    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:40:25.492986 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:25.492815    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:40:35.492997 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:35.492898    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:40:35.493538 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:35.493325    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:40:45.493097 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:45.493053    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:40:45.493621 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:45.493156    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:40:59.701536 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.701500    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"]
May 20 15:40:59.702053 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.701989    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" containerID="cri-o://82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637" gracePeriod=30
May 20 15:40:59.702126 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.702024    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" containerID="cri-o://6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f" gracePeriod=30
May 20 15:40:59.702126 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.702034    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy" containerID="cri-o://4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402" gracePeriod=30
May 20 15:40:59.818247 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.818215    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"]
May 20 15:40:59.818654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.818639    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10d66096-0b67-4cbd-be29-2c08f88d145e" containerName="console"
May 20 15:40:59.818703 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.818656    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="10d66096-0b67-4cbd-be29-2c08f88d145e" containerName="console"
May 20 15:40:59.818737 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.818719    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="10d66096-0b67-4cbd-be29-2c08f88d145e" containerName="console"
May 20 15:40:59.820897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.820879    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:40:59.823765 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.823720    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-61d94-predictor-serving-cert\""
May 20 15:40:59.823765 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.823762    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\""
May 20 15:40:59.833477 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.833447    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"]
May 20 15:40:59.902687 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.902653    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"]
May 20 15:40:59.905313 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.905290    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:40:59.907805 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.907784    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-61d94-predictor-serving-cert\""
May 20 15:40:59.908006 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.907992    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\""
May 20 15:40:59.917008 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.916982    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"]
May 20 15:40:59.960910 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.960787    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3579e22-6597-436b-a1ca-2c6fac6e14a9-proxy-tls\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:40:59.960910 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.960838    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3579e22-6597-436b-a1ca-2c6fac6e14a9-isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:40:59.960910 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.960874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvhb\" (UniqueName: \"kubernetes.io/projected/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kube-api-access-bdvhb\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:40:59.961145 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:40:59.960947    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.062000 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.061956    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-proxy-tls\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.062214 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.062075    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3579e22-6597-436b-a1ca-2c6fac6e14a9-proxy-tls\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.062214 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.062101    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.062214 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.062157    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3579e22-6597-436b-a1ca-2c6fac6e14a9-isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.062357 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.062217    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvhb\" (UniqueName: \"kubernetes.io/projected/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kube-api-access-bdvhb\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.062357 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.062271    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.062357 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.062322    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.062357 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.062352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklhw\" (UniqueName: \"kubernetes.io/projected/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kube-api-access-bklhw\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.062748 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.062723    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.062994 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.062974    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3579e22-6597-436b-a1ca-2c6fac6e14a9-isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.064682 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.064662    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3579e22-6597-436b-a1ca-2c6fac6e14a9-proxy-tls\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.070767 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.070732    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvhb\" (UniqueName: \"kubernetes.io/projected/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kube-api-access-bdvhb\") pod \"isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.134003 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.133962    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:00.163226 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.163187    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.163440 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.163235    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bklhw\" (UniqueName: \"kubernetes.io/projected/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kube-api-access-bklhw\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.163440 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.163289    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-proxy-tls\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.163440 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.163355    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.163804 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.163782    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.163982 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.163960    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.165848 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.165823    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-proxy-tls\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.172356 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.172331    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklhw\" (UniqueName: \"kubernetes.io/projected/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kube-api-access-bklhw\") pod \"isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.216415 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.216301    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:00.283193 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.283164    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"]
May 20 15:41:00.286082 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:41:00.286049    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3579e22_6597_436b_a1ca_2c6fac6e14a9.slice/crio-efda4e12961e3cd902cc7263c6afda8f77f025fae7ab85029ff49a26e617103e WatchSource:0}: Error finding container efda4e12961e3cd902cc7263c6afda8f77f025fae7ab85029ff49a26e617103e: Status 404 returned error can't find the container with id efda4e12961e3cd902cc7263c6afda8f77f025fae7ab85029ff49a26e617103e
May 20 15:41:00.365745 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.365714    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"]
May 20 15:41:00.367677 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:41:00.367646    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbbf461_dd50_48d3_acc8_57d1c14b4c65.slice/crio-8372acf3553a99820066cfa8a80dd3fcacd7ee25453c57bab703d4aa23b6a48d WatchSource:0}: Error finding container 8372acf3553a99820066cfa8a80dd3fcacd7ee25453c57bab703d4aa23b6a48d: Status 404 returned error can't find the container with id 8372acf3553a99820066cfa8a80dd3fcacd7ee25453c57bab703d4aa23b6a48d
May 20 15:41:00.765232 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.765198    2570 generic.go:358] "Generic (PLEG): container finished" podID="28616225-17ab-40a2-8c93-3f711b09517d" containerID="4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402" exitCode=2
May 20 15:41:00.765723 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.765266    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerDied","Data":"4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402"}
May 20 15:41:00.766896 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.766863    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" event={"ID":"0cbbf461-dd50-48d3-acc8-57d1c14b4c65","Type":"ContainerStarted","Data":"88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0"}
May 20 15:41:00.767059 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.766903    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" event={"ID":"0cbbf461-dd50-48d3-acc8-57d1c14b4c65","Type":"ContainerStarted","Data":"8372acf3553a99820066cfa8a80dd3fcacd7ee25453c57bab703d4aa23b6a48d"}
May 20 15:41:00.768429 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.768405    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" event={"ID":"b3579e22-6597-436b-a1ca-2c6fac6e14a9","Type":"ContainerStarted","Data":"bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b"}
May 20 15:41:00.768554 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:00.768434    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" event={"ID":"b3579e22-6597-436b-a1ca-2c6fac6e14a9","Type":"ContainerStarted","Data":"efda4e12961e3cd902cc7263c6afda8f77f025fae7ab85029ff49a26e617103e"}
May 20 15:41:04.489520 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:04.489431    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 15:41:04.784765 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:04.784736    2570 generic.go:358] "Generic (PLEG): container finished" podID="28616225-17ab-40a2-8c93-3f711b09517d" containerID="82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637" exitCode=0
May 20 15:41:04.784932 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:04.784812    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerDied","Data":"82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637"}
May 20 15:41:04.786078 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:04.786056    2570 generic.go:358] "Generic (PLEG): container finished" podID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerID="88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0" exitCode=0
May 20 15:41:04.786156 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:04.786134    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" event={"ID":"0cbbf461-dd50-48d3-acc8-57d1c14b4c65","Type":"ContainerDied","Data":"88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0"}
May 20 15:41:04.790408 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:04.790386    2570 generic.go:358] "Generic (PLEG): container finished" podID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerID="bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b" exitCode=0
May 20 15:41:04.790500 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:04.790436    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" event={"ID":"b3579e22-6597-436b-a1ca-2c6fac6e14a9","Type":"ContainerDied","Data":"bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b"}
May 20 15:41:05.492775 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:05.492731    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:41:05.493223 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:05.493060    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:41:05.797447 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:05.797406    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" event={"ID":"b3579e22-6597-436b-a1ca-2c6fac6e14a9","Type":"ContainerStarted","Data":"b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f"}
May 20 15:41:05.797648 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:05.797462    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" event={"ID":"b3579e22-6597-436b-a1ca-2c6fac6e14a9","Type":"ContainerStarted","Data":"730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75"}
May 20 15:41:05.797926 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:05.797889    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:05.818649 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:05.818531    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podStartSLOduration=6.818502668 podStartE2EDuration="6.818502668s" podCreationTimestamp="2026-05-20 15:40:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:41:05.817280312 +0000 UTC m=+554.647376234" watchObservedRunningTime="2026-05-20 15:41:05.818502668 +0000 UTC m=+554.648598589"
May 20 15:41:06.801838 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:06.801794    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:06.803474 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:06.803428    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:41:07.806093 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:07.806052    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:41:09.489740 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:09.489688    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 15:41:12.811214 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:12.811172    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:41:12.811844 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:12.811817    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:41:14.489540 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:14.489493    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 15:41:14.490003 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:14.489665    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:41:15.493132 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:15.492928    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:41:15.493624 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:15.493312    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:41:19.489888 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:19.489832    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 15:41:22.812383 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:22.812322    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:41:24.489230 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:24.489167    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 15:41:25.492543 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:25.492483    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
May 20 15:41:25.493091 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:25.492694    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:41:25.493091 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:25.492844    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:41:25.493091 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:25.492957    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:41:25.876852 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:25.876819    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" event={"ID":"0cbbf461-dd50-48d3-acc8-57d1c14b4c65","Type":"ContainerStarted","Data":"40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2"}
May 20 15:41:26.883796 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:26.883755    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" event={"ID":"0cbbf461-dd50-48d3-acc8-57d1c14b4c65","Type":"ContainerStarted","Data":"41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d"}
May 20 15:41:26.884197 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:26.883910    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:26.906075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:26.906011    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podStartSLOduration=6.93023085 podStartE2EDuration="27.905990403s" podCreationTimestamp="2026-05-20 15:40:59 +0000 UTC" firstStartedPulling="2026-05-20 15:41:04.788154591 +0000 UTC m=+553.618250492" lastFinishedPulling="2026-05-20 15:41:25.763914144 +0000 UTC m=+574.594010045" observedRunningTime="2026-05-20 15:41:26.90432512 +0000 UTC m=+575.734421049" watchObservedRunningTime="2026-05-20 15:41:26.905990403 +0000 UTC m=+575.736086325"
May 20 15:41:27.886937 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:27.886899    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:27.888082 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:27.888055    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 15:41:28.890622 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:28.890585    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 15:41:29.489303 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.489259    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 15:41:29.851955 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.851928    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:41:29.895641 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.895606    2570 generic.go:358] "Generic (PLEG): container finished" podID="28616225-17ab-40a2-8c93-3f711b09517d" containerID="6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f" exitCode=0
May 20 15:41:29.896036 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.895682    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerDied","Data":"6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f"}
May 20 15:41:29.896036 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.895718    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"
May 20 15:41:29.896036 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.895734    2570 scope.go:117] "RemoveContainer" containerID="6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f"
May 20 15:41:29.896036 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.895723    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw" event={"ID":"28616225-17ab-40a2-8c93-3f711b09517d","Type":"ContainerDied","Data":"f256893216aa8769e56e4af2228fc05155a7cd6ddae69b5f2661d8412d8d997f"}
May 20 15:41:29.903959 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.903941    2570 scope.go:117] "RemoveContainer" containerID="4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402"
May 20 15:41:29.912339 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.912315    2570 scope.go:117] "RemoveContainer" containerID="82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637"
May 20 15:41:29.920666 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.920646    2570 scope.go:117] "RemoveContainer" containerID="8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5"
May 20 15:41:29.928680 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.928652    2570 scope.go:117] "RemoveContainer" containerID="6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f"
May 20 15:41:29.928991 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:41:29.928968    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f\": container with ID starting with 6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f not found: ID does not exist" containerID="6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f"
May 20 15:41:29.929042 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.929002    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f"} err="failed to get container status \"6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f\": rpc error: code = NotFound desc = could not find container \"6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f\": container with ID starting with 6e5b5f59b486a03c2078cc37fc84e07e8d696b97d3ba4ecbd19570c579a2a38f not found: ID does not exist"
May 20 15:41:29.929042 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.929023    2570 scope.go:117] "RemoveContainer" containerID="4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402"
May 20 15:41:29.929294 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:41:29.929276    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402\": container with ID starting with 4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402 not found: ID does not exist" containerID="4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402"
May 20 15:41:29.929387 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.929300    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402"} err="failed to get container status \"4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402\": rpc error: code = NotFound desc = could not find container \"4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402\": container with ID starting with 4db615254c1e7aa91efc48cc8d18fde5bdb9b8badc1f3daa847e61029a605402 not found: ID does not exist"
May 20 15:41:29.929387 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.929316    2570 scope.go:117] "RemoveContainer" containerID="82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637"
May 20 15:41:29.929547 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:41:29.929526    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637\": container with ID starting with 82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637 not found: ID does not exist" containerID="82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637"
May 20 15:41:29.929595 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.929551    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637"} err="failed to get container status \"82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637\": rpc error: code = NotFound desc = could not find container \"82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637\": container with ID starting with 82109c1e4426e4f7178037299cad00031a792523648dad2d7c61a7f7cc83c637 not found: ID does not exist"
May 20 15:41:29.929595 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.929566    2570 scope.go:117] "RemoveContainer" containerID="8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5"
May 20 15:41:29.929772 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:41:29.929748    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5\": container with ID starting with 8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5 not found: ID does not exist" containerID="8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5"
May 20 15:41:29.929854 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.929772    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5"} err="failed to get container status \"8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5\": rpc error: code = NotFound desc = could not find container \"8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5\": container with ID starting with 8ec95729d5ba6972346dfb7a300201b096fb4e79562d550cc076e7749b9633d5 not found: ID does not exist"
May 20 15:41:29.959150 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.959115    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28616225-17ab-40a2-8c93-3f711b09517d-kserve-provision-location\") pod \"28616225-17ab-40a2-8c93-3f711b09517d\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") "
May 20 15:41:29.959330 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.959226    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28616225-17ab-40a2-8c93-3f711b09517d-proxy-tls\") pod \"28616225-17ab-40a2-8c93-3f711b09517d\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") "
May 20 15:41:29.959330 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.959261    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28616225-17ab-40a2-8c93-3f711b09517d-isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\") pod \"28616225-17ab-40a2-8c93-3f711b09517d\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") "
May 20 15:41:29.959330 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.959279    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gff49\" (UniqueName: \"kubernetes.io/projected/28616225-17ab-40a2-8c93-3f711b09517d-kube-api-access-gff49\") pod \"28616225-17ab-40a2-8c93-3f711b09517d\" (UID: \"28616225-17ab-40a2-8c93-3f711b09517d\") "
May 20 15:41:29.959501 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.959466    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28616225-17ab-40a2-8c93-3f711b09517d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28616225-17ab-40a2-8c93-3f711b09517d" (UID: "28616225-17ab-40a2-8c93-3f711b09517d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:41:29.959722 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.959693    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28616225-17ab-40a2-8c93-3f711b09517d-isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config") pod "28616225-17ab-40a2-8c93-3f711b09517d" (UID: "28616225-17ab-40a2-8c93-3f711b09517d"). InnerVolumeSpecName "isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:41:29.961575 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.961554    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28616225-17ab-40a2-8c93-3f711b09517d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28616225-17ab-40a2-8c93-3f711b09517d" (UID: "28616225-17ab-40a2-8c93-3f711b09517d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:41:29.961680 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:29.961576    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28616225-17ab-40a2-8c93-3f711b09517d-kube-api-access-gff49" (OuterVolumeSpecName: "kube-api-access-gff49") pod "28616225-17ab-40a2-8c93-3f711b09517d" (UID: "28616225-17ab-40a2-8c93-3f711b09517d"). InnerVolumeSpecName "kube-api-access-gff49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:41:30.060803 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:30.060763    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28616225-17ab-40a2-8c93-3f711b09517d-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:41:30.060803 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:30.060795    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28616225-17ab-40a2-8c93-3f711b09517d-isvc-raw-sklearn-batcher-be7d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:41:30.060803 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:30.060808    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gff49\" (UniqueName: \"kubernetes.io/projected/28616225-17ab-40a2-8c93-3f711b09517d-kube-api-access-gff49\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:41:30.061033 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:30.060818    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28616225-17ab-40a2-8c93-3f711b09517d-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:41:30.218926 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:30.218882    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"]
May 20 15:41:30.222119 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:30.222092    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-be7d1-predictor-d8667789c-j4mxw"]
May 20 15:41:31.759759 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:31.759721    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28616225-17ab-40a2-8c93-3f711b09517d" path="/var/lib/kubelet/pods/28616225-17ab-40a2-8c93-3f711b09517d/volumes"
May 20 15:41:32.812238 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:32.812197    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:41:33.895759 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:33.895732    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:41:33.896341 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:33.896313    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 15:41:42.811662 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:42.811622    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:41:43.896657 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:43.896617    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 15:41:51.666208 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:51.666182    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:41:51.666683 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:51.666598    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:41:52.812224 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:52.812181    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:41:53.896919 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:41:53.896878    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 15:42:02.812183 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:02.812136    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:42:03.896270 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:03.896225    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 15:42:12.812469 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:12.812353    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:42:13.896530 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:13.896489    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 15:42:22.812503 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:22.812469    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:42:23.897065 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:23.897021    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 15:42:33.897409 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:33.897348    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:42:50.195245 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.195207    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"]
May 20 15:42:50.195792 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.195626    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" containerID="cri-o://730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75" gracePeriod=30
May 20 15:42:50.195792 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.195663    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kube-rbac-proxy" containerID="cri-o://b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f" gracePeriod=30
May 20 15:42:50.281216 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281179    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"]
May 20 15:42:50.281602 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281587    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container"
May 20 15:42:50.281654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281604    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container"
May 20 15:42:50.281654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281612    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy"
May 20 15:42:50.281654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281618    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy"
May 20 15:42:50.281654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281630    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent"
May 20 15:42:50.281654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281635    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent"
May 20 15:42:50.281654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281647    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="storage-initializer"
May 20 15:42:50.281654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281652    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="storage-initializer"
May 20 15:42:50.281887 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281714    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kube-rbac-proxy"
May 20 15:42:50.281887 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281726    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="kserve-container"
May 20 15:42:50.281887 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.281740    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="28616225-17ab-40a2-8c93-3f711b09517d" containerName="agent"
May 20 15:42:50.285143 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.285125    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.287786 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.287755    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-86fe3-predictor-serving-cert\""
May 20 15:42:50.287925 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.287755    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\""
May 20 15:42:50.297027 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.296994    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"]
May 20 15:42:50.360047 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.360009    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"]
May 20 15:42:50.363756 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.363727    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.366652 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.366629    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-86fe3-predictor-serving-cert\""
May 20 15:42:50.366795 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.366705    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\""
May 20 15:42:50.374124 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.374097    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"]
May 20 15:42:50.396050 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.396007    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1f60d1-cf8f-4598-982d-9faea05bb01a-isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.396255 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.396088    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4r46\" (UniqueName: \"kubernetes.io/projected/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kube-api-access-p4r46\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.396255 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.396134    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1f60d1-cf8f-4598-982d-9faea05bb01a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.396255 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.396162    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.420307 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.420269    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"]
May 20 15:42:50.420682 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.420635    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" containerID="cri-o://40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2" gracePeriod=30
May 20 15:42:50.420828 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.420673    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kube-rbac-proxy" containerID="cri-o://41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d" gracePeriod=30
May 20 15:42:50.497130 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497033    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1f60d1-cf8f-4598-982d-9faea05bb01a-isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.497130 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497088    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.497360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497128    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p4r46\" (UniqueName: \"kubernetes.io/projected/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kube-api-access-p4r46\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.497360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497167    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1f60d1-cf8f-4598-982d-9faea05bb01a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.497360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497201    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.497360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mbv\" (UniqueName: \"kubernetes.io/projected/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kube-api-access-d6mbv\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.497360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497295    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.497360 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:42:50.497306    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-86fe3-predictor-serving-cert" not found
May 20 15:42:50.497742 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:42:50.497411    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1f60d1-cf8f-4598-982d-9faea05bb01a-proxy-tls podName:bf1f60d1-cf8f-4598-982d-9faea05bb01a nodeName:}" failed. No retries permitted until 2026-05-20 15:42:50.997352708 +0000 UTC m=+659.827448622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf1f60d1-cf8f-4598-982d-9faea05bb01a-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" (UID: "bf1f60d1-cf8f-4598-982d-9faea05bb01a") : secret "isvc-sklearn-graph-raw-hpa-86fe3-predictor-serving-cert" not found
May 20 15:42:50.497742 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497440    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.497742 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497651    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.497840 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.497777    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1f60d1-cf8f-4598-982d-9faea05bb01a-isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.514707 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.514671    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4r46\" (UniqueName: \"kubernetes.io/projected/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kube-api-access-p4r46\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:50.597932 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.597885    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.598133 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.597999    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mbv\" (UniqueName: \"kubernetes.io/projected/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kube-api-access-d6mbv\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.598133 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.598026    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.598133 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.598060    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.598506 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.598476    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.598807 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.598775    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.600615 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.600589    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.609891 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.609856    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mbv\" (UniqueName: \"kubernetes.io/projected/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kube-api-access-d6mbv\") pod \"isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.675671 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.675627    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:50.830644 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:50.830615    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"]
May 20 15:42:50.834012 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:42:50.833934    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b36bd82_8eeb_454e_9ef7_6d85bc801e0a.slice/crio-835587e215034d3d277de0fc9e88cef5334d5471f4abb65821665a1d9df237a1 WatchSource:0}: Error finding container 835587e215034d3d277de0fc9e88cef5334d5471f4abb65821665a1d9df237a1: Status 404 returned error can't find the container with id 835587e215034d3d277de0fc9e88cef5334d5471f4abb65821665a1d9df237a1
May 20 15:42:51.002147 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.002093    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1f60d1-cf8f-4598-982d-9faea05bb01a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:51.004617 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.004587    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1f60d1-cf8f-4598-982d-9faea05bb01a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:51.190124 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.190089    2570 generic.go:358] "Generic (PLEG): container finished" podID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerID="41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d" exitCode=2
May 20 15:42:51.190321 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.190164    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" event={"ID":"0cbbf461-dd50-48d3-acc8-57d1c14b4c65","Type":"ContainerDied","Data":"41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d"}
May 20 15:42:51.191580 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.191548    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" event={"ID":"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a","Type":"ContainerStarted","Data":"471909633eb241c42942bd843a0ffc46f595018f77857c56a687fbca3ef55e5a"}
May 20 15:42:51.191720 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.191584    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" event={"ID":"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a","Type":"ContainerStarted","Data":"835587e215034d3d277de0fc9e88cef5334d5471f4abb65821665a1d9df237a1"}
May 20 15:42:51.193264 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.193241    2570 generic.go:358] "Generic (PLEG): container finished" podID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerID="b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f" exitCode=2
May 20 15:42:51.193354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.193311    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" event={"ID":"b3579e22-6597-436b-a1ca-2c6fac6e14a9","Type":"ContainerDied","Data":"b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f"}
May 20 15:42:51.196449 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.196428    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:51.334777 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:51.334736    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"]
May 20 15:42:51.337843 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:42:51.337809    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1f60d1_cf8f_4598_982d_9faea05bb01a.slice/crio-eb0359b695e576189a98dc079198ce3d3b762490095d5dc938d65cdf73781f63 WatchSource:0}: Error finding container eb0359b695e576189a98dc079198ce3d3b762490095d5dc938d65cdf73781f63: Status 404 returned error can't find the container with id eb0359b695e576189a98dc079198ce3d3b762490095d5dc938d65cdf73781f63
May 20 15:42:52.198310 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:52.198272    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" event={"ID":"bf1f60d1-cf8f-4598-982d-9faea05bb01a","Type":"ContainerStarted","Data":"5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6"}
May 20 15:42:52.198310 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:52.198311    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" event={"ID":"bf1f60d1-cf8f-4598-982d-9faea05bb01a","Type":"ContainerStarted","Data":"eb0359b695e576189a98dc079198ce3d3b762490095d5dc938d65cdf73781f63"}
May 20 15:42:52.806747 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:52.806706    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
May 20 15:42:52.812104 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:52.812081    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 15:42:53.891162 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:53.891117    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
May 20 15:42:53.896595 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:53.896568    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 15:42:54.160816 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.160794    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:42:54.207848 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.207815    2570 generic.go:358] "Generic (PLEG): container finished" podID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerID="40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2" exitCode=0
May 20 15:42:54.207956 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.207884    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"
May 20 15:42:54.207956 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.207891    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" event={"ID":"0cbbf461-dd50-48d3-acc8-57d1c14b4c65","Type":"ContainerDied","Data":"40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2"}
May 20 15:42:54.207956 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.207925    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq" event={"ID":"0cbbf461-dd50-48d3-acc8-57d1c14b4c65","Type":"ContainerDied","Data":"8372acf3553a99820066cfa8a80dd3fcacd7ee25453c57bab703d4aa23b6a48d"}
May 20 15:42:54.207956 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.207942    2570 scope.go:117] "RemoveContainer" containerID="41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d"
May 20 15:42:54.217458 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.217436    2570 scope.go:117] "RemoveContainer" containerID="40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2"
May 20 15:42:54.224373 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.224345    2570 scope.go:117] "RemoveContainer" containerID="88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0"
May 20 15:42:54.231080 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.231062    2570 scope.go:117] "RemoveContainer" containerID="41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d"
May 20 15:42:54.231314 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:42:54.231297    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d\": container with ID starting with 41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d not found: ID does not exist" containerID="41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d"
May 20 15:42:54.231360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.231325    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d"} err="failed to get container status \"41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d\": rpc error: code = NotFound desc = could not find container \"41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d\": container with ID starting with 41f7e536a691b7143a4ed99b342d961effeb45256ecb8b33f5605a619f00be3d not found: ID does not exist"
May 20 15:42:54.231360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.231343    2570 scope.go:117] "RemoveContainer" containerID="40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2"
May 20 15:42:54.231592 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:42:54.231575    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2\": container with ID starting with 40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2 not found: ID does not exist" containerID="40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2"
May 20 15:42:54.231630 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.231601    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2"} err="failed to get container status \"40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2\": rpc error: code = NotFound desc = could not find container \"40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2\": container with ID starting with 40a9c3da9b73aeda6561d1f1a7eb245dcc3d081ead0a520a3cc3b0d5f34363a2 not found: ID does not exist"
May 20 15:42:54.231630 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.231618    2570 scope.go:117] "RemoveContainer" containerID="88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0"
May 20 15:42:54.231823 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:42:54.231808    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0\": container with ID starting with 88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0 not found: ID does not exist" containerID="88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0"
May 20 15:42:54.231865 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.231829    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0"} err="failed to get container status \"88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0\": rpc error: code = NotFound desc = could not find container \"88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0\": container with ID starting with 88bcd2cfc0eb86426d956c75230f8d6dc1a5e4bb16b25e514b356be0594fcda0 not found: ID does not exist"
May 20 15:42:54.233060 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.233043    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\") pod \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") "
May 20 15:42:54.233098 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.233080    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bklhw\" (UniqueName: \"kubernetes.io/projected/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kube-api-access-bklhw\") pod \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") "
May 20 15:42:54.233148 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.233127    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kserve-provision-location\") pod \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") "
May 20 15:42:54.233217 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.233206    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-proxy-tls\") pod \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\" (UID: \"0cbbf461-dd50-48d3-acc8-57d1c14b4c65\") "
May 20 15:42:54.233450 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.233427    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config") pod "0cbbf461-dd50-48d3-acc8-57d1c14b4c65" (UID: "0cbbf461-dd50-48d3-acc8-57d1c14b4c65"). InnerVolumeSpecName "isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:42:54.233450 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.233437    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0cbbf461-dd50-48d3-acc8-57d1c14b4c65" (UID: "0cbbf461-dd50-48d3-acc8-57d1c14b4c65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:54.235089 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.235064    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0cbbf461-dd50-48d3-acc8-57d1c14b4c65" (UID: "0cbbf461-dd50-48d3-acc8-57d1c14b4c65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:42:54.235187 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.235142    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kube-api-access-bklhw" (OuterVolumeSpecName: "kube-api-access-bklhw") pod "0cbbf461-dd50-48d3-acc8-57d1c14b4c65" (UID: "0cbbf461-dd50-48d3-acc8-57d1c14b4c65"). InnerVolumeSpecName "kube-api-access-bklhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:42:54.334725 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.334698    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-isvc-xgboost-graph-raw-61d94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:42:54.334725 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.334723    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bklhw\" (UniqueName: \"kubernetes.io/projected/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kube-api-access-bklhw\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:42:54.334861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.334733    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:42:54.334861 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.334742    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbbf461-dd50-48d3-acc8-57d1c14b4c65-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:42:54.535207 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.535170    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"]
May 20 15:42:54.538257 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:54.538233    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-61d94-predictor-66c75f745c-4pgkq"]
May 20 15:42:55.215280 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:55.214807    2570 generic.go:358] "Generic (PLEG): container finished" podID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerID="5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6" exitCode=0
May 20 15:42:55.215280 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:55.214931    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" event={"ID":"bf1f60d1-cf8f-4598-982d-9faea05bb01a","Type":"ContainerDied","Data":"5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6"}
May 20 15:42:55.218329 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:55.218305    2570 generic.go:358] "Generic (PLEG): container finished" podID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerID="471909633eb241c42942bd843a0ffc46f595018f77857c56a687fbca3ef55e5a" exitCode=0
May 20 15:42:55.218455 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:55.218391    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" event={"ID":"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a","Type":"ContainerDied","Data":"471909633eb241c42942bd843a0ffc46f595018f77857c56a687fbca3ef55e5a"}
May 20 15:42:55.760468 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:55.760435    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" path="/var/lib/kubelet/pods/0cbbf461-dd50-48d3-acc8-57d1c14b4c65/volumes"
May 20 15:42:56.223428 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.223393    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" event={"ID":"bf1f60d1-cf8f-4598-982d-9faea05bb01a","Type":"ContainerStarted","Data":"8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9"}
May 20 15:42:56.223866 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.223440    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" event={"ID":"bf1f60d1-cf8f-4598-982d-9faea05bb01a","Type":"ContainerStarted","Data":"b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488"}
May 20 15:42:56.223866 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.223731    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:56.223969 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.223863    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:42:56.224825 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.224797    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 15:42:56.225352 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.225333    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" event={"ID":"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a","Type":"ContainerStarted","Data":"b7c9603fbdefadd38f3e5a1dfecb61f24c997fb2393701959f9297d4dedac288"}
May 20 15:42:56.225465 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.225358    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" event={"ID":"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a","Type":"ContainerStarted","Data":"f87003359bcce86ce6fa490d09d8dffe764b924c4b2000a948a1b562d4f813f3"}
May 20 15:42:56.225569 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.225552    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:56.244198 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.244151    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podStartSLOduration=6.244136032 podStartE2EDuration="6.244136032s" podCreationTimestamp="2026-05-20 15:42:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:42:56.243565163 +0000 UTC m=+665.073661085" watchObservedRunningTime="2026-05-20 15:42:56.244136032 +0000 UTC m=+665.074231958"
May 20 15:42:56.265432 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:56.265385    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podStartSLOduration=6.265356772 podStartE2EDuration="6.265356772s" podCreationTimestamp="2026-05-20 15:42:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:42:56.263266754 +0000 UTC m=+665.093362697" watchObservedRunningTime="2026-05-20 15:42:56.265356772 +0000 UTC m=+665.095452693"
May 20 15:42:57.229382 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:57.229336    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 15:42:57.229762 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:57.229663    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:42:57.230593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:57.230567    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 15:42:57.806566 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:57.806532    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
May 20 15:42:57.942984 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:57.942960    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:42:58.069998 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.069966    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdvhb\" (UniqueName: \"kubernetes.io/projected/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kube-api-access-bdvhb\") pod \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") "
May 20 15:42:58.070150 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.070027    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kserve-provision-location\") pod \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") "
May 20 15:42:58.070150 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.070054    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3579e22-6597-436b-a1ca-2c6fac6e14a9-isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\") pod \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") "
May 20 15:42:58.070150 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.070099    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3579e22-6597-436b-a1ca-2c6fac6e14a9-proxy-tls\") pod \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\" (UID: \"b3579e22-6597-436b-a1ca-2c6fac6e14a9\") "
May 20 15:42:58.070462 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.070431    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3579e22-6597-436b-a1ca-2c6fac6e14a9" (UID: "b3579e22-6597-436b-a1ca-2c6fac6e14a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:58.070462 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.070451    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3579e22-6597-436b-a1ca-2c6fac6e14a9-isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config") pod "b3579e22-6597-436b-a1ca-2c6fac6e14a9" (UID: "b3579e22-6597-436b-a1ca-2c6fac6e14a9"). InnerVolumeSpecName "isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:42:58.071899 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.071879    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kube-api-access-bdvhb" (OuterVolumeSpecName: "kube-api-access-bdvhb") pod "b3579e22-6597-436b-a1ca-2c6fac6e14a9" (UID: "b3579e22-6597-436b-a1ca-2c6fac6e14a9"). InnerVolumeSpecName "kube-api-access-bdvhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:42:58.071980 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.071963    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3579e22-6597-436b-a1ca-2c6fac6e14a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3579e22-6597-436b-a1ca-2c6fac6e14a9" (UID: "b3579e22-6597-436b-a1ca-2c6fac6e14a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:42:58.170620 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.170592    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3579e22-6597-436b-a1ca-2c6fac6e14a9-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:42:58.170620 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.170618    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdvhb\" (UniqueName: \"kubernetes.io/projected/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kube-api-access-bdvhb\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:42:58.170759 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.170633    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3579e22-6597-436b-a1ca-2c6fac6e14a9-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:42:58.170759 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.170646    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3579e22-6597-436b-a1ca-2c6fac6e14a9-isvc-sklearn-graph-raw-61d94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:42:58.234870 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.234840    2570 generic.go:358] "Generic (PLEG): container finished" podID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerID="730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75" exitCode=0
May 20 15:42:58.235227 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.234912    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"
May 20 15:42:58.235227 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.234917    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" event={"ID":"b3579e22-6597-436b-a1ca-2c6fac6e14a9","Type":"ContainerDied","Data":"730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75"}
May 20 15:42:58.235227 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.234951    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4" event={"ID":"b3579e22-6597-436b-a1ca-2c6fac6e14a9","Type":"ContainerDied","Data":"efda4e12961e3cd902cc7263c6afda8f77f025fae7ab85029ff49a26e617103e"}
May 20 15:42:58.235227 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.234966    2570 scope.go:117] "RemoveContainer" containerID="b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f"
May 20 15:42:58.235618 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.235591    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 15:42:58.244086 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.244069    2570 scope.go:117] "RemoveContainer" containerID="730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75"
May 20 15:42:58.251346 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.251331    2570 scope.go:117] "RemoveContainer" containerID="bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b"
May 20 15:42:58.257287 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.257266    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"]
May 20 15:42:58.258820 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.258777    2570 scope.go:117] "RemoveContainer" containerID="b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f"
May 20 15:42:58.259099 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:42:58.259069    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f\": container with ID starting with b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f not found: ID does not exist" containerID="b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f"
May 20 15:42:58.259187 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.259104    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f"} err="failed to get container status \"b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f\": rpc error: code = NotFound desc = could not find container \"b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f\": container with ID starting with b901d9ec1505d629d45843fb6dfe3d6808d1dd47cf8d02441b2ab88a4b66883f not found: ID does not exist"
May 20 15:42:58.259187 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.259127    2570 scope.go:117] "RemoveContainer" containerID="730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75"
May 20 15:42:58.259826 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:42:58.259803    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75\": container with ID starting with 730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75 not found: ID does not exist" containerID="730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75"
May 20 15:42:58.259906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.259833    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75"} err="failed to get container status \"730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75\": rpc error: code = NotFound desc = could not find container \"730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75\": container with ID starting with 730d32abe4e629596cea6ebd510fcf41ebb49e325323423a94e8bd40cb9c5e75 not found: ID does not exist"
May 20 15:42:58.259906 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.259850    2570 scope.go:117] "RemoveContainer" containerID="bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b"
May 20 15:42:58.260090 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:42:58.260073    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b\": container with ID starting with bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b not found: ID does not exist" containerID="bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b"
May 20 15:42:58.260137 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.260098    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b"} err="failed to get container status \"bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b\": rpc error: code = NotFound desc = could not find container \"bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b\": container with ID starting with bd503904eb84e3003d02720cc1589efb0069a4740a85e73666c4390e16a3303b not found: ID does not exist"
May 20 15:42:58.261544 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:58.261524    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-61d94-predictor-794dddc95c-frrv4"]
May 20 15:42:59.759979 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:42:59.759943    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" path="/var/lib/kubelet/pods/b3579e22-6597-436b-a1ca-2c6fac6e14a9/volumes"
May 20 15:43:02.234064 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:02.234036    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:43:02.234706 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:02.234681    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 15:43:03.241220 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:03.241189    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:43:03.241867 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:03.241837    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 15:43:12.235636 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:12.235598    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 15:43:13.242350 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:13.242307    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 15:43:22.235430 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:22.235388    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 15:43:23.242719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:23.242680    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 15:43:32.235175 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:32.235130    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 15:43:33.241939 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:33.241892    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 15:43:42.234972 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:42.234886    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 15:43:43.242166 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:43.242120    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 15:43:52.234557 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:52.234513    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 15:43:53.242414 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:43:53.242361    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 15:44:02.236042 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:02.235995    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:44:03.242523 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:03.242493    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:44:30.482699 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.482661    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"]
May 20 15:44:30.483109 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.482987    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" containerID="cri-o://b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488" gracePeriod=30
May 20 15:44:30.483109 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.483047    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kube-rbac-proxy" containerID="cri-o://8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9" gracePeriod=30
May 20 15:44:30.546705 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.546666    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"]
May 20 15:44:30.547048 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547037    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="storage-initializer"
May 20 15:44:30.547101 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547050    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="storage-initializer"
May 20 15:44:30.547101 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547069    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kube-rbac-proxy"
May 20 15:44:30.547101 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547074    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kube-rbac-proxy"
May 20 15:44:30.547101 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547089    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="storage-initializer"
May 20 15:44:30.547101 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547095    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="storage-initializer"
May 20 15:44:30.547101 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547103    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container"
May 20 15:44:30.547273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547108    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container"
May 20 15:44:30.547273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547122    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kube-rbac-proxy"
May 20 15:44:30.547273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547127    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kube-rbac-proxy"
May 20 15:44:30.547273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547135    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container"
May 20 15:44:30.547273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547140    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container"
May 20 15:44:30.547273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547196    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kube-rbac-proxy"
May 20 15:44:30.547273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547209    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kserve-container"
May 20 15:44:30.547273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547216    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="b3579e22-6597-436b-a1ca-2c6fac6e14a9" containerName="kube-rbac-proxy"
May 20 15:44:30.547273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.547222    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="0cbbf461-dd50-48d3-acc8-57d1c14b4c65" containerName="kserve-container"
May 20 15:44:30.550793 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.550769    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.553654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.553629    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-47eff-predictor-serving-cert\""
May 20 15:44:30.553935 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.553917    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-47eff-kube-rbac-proxy-sar-config\""
May 20 15:44:30.572309 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.572277    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"]
May 20 15:44:30.622067 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.622021    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"]
May 20 15:44:30.622510 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.622467    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" containerID="cri-o://f87003359bcce86ce6fa490d09d8dffe764b924c4b2000a948a1b562d4f813f3" gracePeriod=30
May 20 15:44:30.622581 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.622492    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kube-rbac-proxy" containerID="cri-o://b7c9603fbdefadd38f3e5a1dfecb61f24c997fb2393701959f9297d4dedac288" gracePeriod=30
May 20 15:44:30.719404 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.719344    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc63f01a-f82a-4ef8-b174-83adb94e1fad-proxy-tls\") pod \"message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") " pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.719594 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.719462    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxrh\" (UniqueName: \"kubernetes.io/projected/bc63f01a-f82a-4ef8-b174-83adb94e1fad-kube-api-access-9qxrh\") pod \"message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") " pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.719594 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.719529    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc63f01a-f82a-4ef8-b174-83adb94e1fad-message-dumper-raw-47eff-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") " pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.821030 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.820992    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc63f01a-f82a-4ef8-b174-83adb94e1fad-proxy-tls\") pod \"message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") " pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.821242 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.821049    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxrh\" (UniqueName: \"kubernetes.io/projected/bc63f01a-f82a-4ef8-b174-83adb94e1fad-kube-api-access-9qxrh\") pod \"message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") " pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.821242 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.821122    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc63f01a-f82a-4ef8-b174-83adb94e1fad-message-dumper-raw-47eff-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") " pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.821825 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.821800    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc63f01a-f82a-4ef8-b174-83adb94e1fad-message-dumper-raw-47eff-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") " pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.823572 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.823550    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc63f01a-f82a-4ef8-b174-83adb94e1fad-proxy-tls\") pod \"message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") " pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.831508 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.831478    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxrh\" (UniqueName: \"kubernetes.io/projected/bc63f01a-f82a-4ef8-b174-83adb94e1fad-kube-api-access-9qxrh\") pod \"message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") " pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:30.861489 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:30.861450    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:31.002630 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:31.002481    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"]
May 20 15:44:31.005274 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:44:31.005241    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc63f01a_f82a_4ef8_b174_83adb94e1fad.slice/crio-6a5d4cef6048c0f4eb705a11c79f356082e6bb98ab8c76bb43c422a871dc2b02 WatchSource:0}: Error finding container 6a5d4cef6048c0f4eb705a11c79f356082e6bb98ab8c76bb43c422a871dc2b02: Status 404 returned error can't find the container with id 6a5d4cef6048c0f4eb705a11c79f356082e6bb98ab8c76bb43c422a871dc2b02
May 20 15:44:31.007077 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:31.007058    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:44:31.558757 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:31.558724    2570 generic.go:358] "Generic (PLEG): container finished" podID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerID="8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9" exitCode=2
May 20 15:44:31.559212 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:31.558795    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" event={"ID":"bf1f60d1-cf8f-4598-982d-9faea05bb01a","Type":"ContainerDied","Data":"8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9"}
May 20 15:44:31.560626 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:31.560603    2570 generic.go:358] "Generic (PLEG): container finished" podID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerID="b7c9603fbdefadd38f3e5a1dfecb61f24c997fb2393701959f9297d4dedac288" exitCode=2
May 20 15:44:31.560751 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:31.560673    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" event={"ID":"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a","Type":"ContainerDied","Data":"b7c9603fbdefadd38f3e5a1dfecb61f24c997fb2393701959f9297d4dedac288"}
May 20 15:44:31.561683 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:31.561665    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc" event={"ID":"bc63f01a-f82a-4ef8-b174-83adb94e1fad","Type":"ContainerStarted","Data":"6a5d4cef6048c0f4eb705a11c79f356082e6bb98ab8c76bb43c422a871dc2b02"}
May 20 15:44:32.229676 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:32.229638    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
May 20 15:44:32.235241 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:32.235208    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
May 20 15:44:32.567204 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:32.567162    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc" event={"ID":"bc63f01a-f82a-4ef8-b174-83adb94e1fad","Type":"ContainerStarted","Data":"48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60"}
May 20 15:44:32.567204 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:32.567209    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc" event={"ID":"bc63f01a-f82a-4ef8-b174-83adb94e1fad","Type":"ContainerStarted","Data":"b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7"}
May 20 15:44:32.567679 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:32.567436    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:32.567679 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:32.567467    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:32.569557 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:32.569533    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:32.592733 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:32.592670    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc" podStartSLOduration=1.491692537 podStartE2EDuration="2.592653103s" podCreationTimestamp="2026-05-20 15:44:30 +0000 UTC" firstStartedPulling="2026-05-20 15:44:31.007199052 +0000 UTC m=+759.837294965" lastFinishedPulling="2026-05-20 15:44:32.108159624 +0000 UTC m=+760.938255531" observedRunningTime="2026-05-20 15:44:32.589829549 +0000 UTC m=+761.419925471" watchObservedRunningTime="2026-05-20 15:44:32.592653103 +0000 UTC m=+761.422749024"
May 20 15:44:33.235975 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:33.235926    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
May 20 15:44:33.241776 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:33.241732    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 15:44:34.576565 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.576527    2570 generic.go:358] "Generic (PLEG): container finished" podID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerID="f87003359bcce86ce6fa490d09d8dffe764b924c4b2000a948a1b562d4f813f3" exitCode=0
May 20 15:44:34.576989 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.576606    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" event={"ID":"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a","Type":"ContainerDied","Data":"f87003359bcce86ce6fa490d09d8dffe764b924c4b2000a948a1b562d4f813f3"}
May 20 15:44:34.670465 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.670440    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:44:34.759268 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.759172    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6mbv\" (UniqueName: \"kubernetes.io/projected/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kube-api-access-d6mbv\") pod \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") "
May 20 15:44:34.759437 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.759279    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kserve-provision-location\") pod \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") "
May 20 15:44:34.759437 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.759307    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-proxy-tls\") pod \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") "
May 20 15:44:34.759437 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.759334    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") pod \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\" (UID: \"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a\") "
May 20 15:44:34.759654 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.759621    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" (UID: "0b36bd82-8eeb-454e-9ef7-6d85bc801e0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:44:34.759819 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.759789    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config") pod "0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" (UID: "0b36bd82-8eeb-454e-9ef7-6d85bc801e0a"). InnerVolumeSpecName "isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:44:34.761439 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.761415    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" (UID: "0b36bd82-8eeb-454e-9ef7-6d85bc801e0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:44:34.761439 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.761418    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kube-api-access-d6mbv" (OuterVolumeSpecName: "kube-api-access-d6mbv") pod "0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" (UID: "0b36bd82-8eeb-454e-9ef7-6d85bc801e0a"). InnerVolumeSpecName "kube-api-access-d6mbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:44:34.861074 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.861005    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:44:34.861074 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.861064    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:44:34.861074 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.861075    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-isvc-xgboost-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:44:34.861074 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:34.861087    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6mbv\" (UniqueName: \"kubernetes.io/projected/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a-kube-api-access-d6mbv\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:44:35.427796 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.427769    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:44:35.568278 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.568239    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4r46\" (UniqueName: \"kubernetes.io/projected/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kube-api-access-p4r46\") pod \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") "
May 20 15:44:35.568509 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.568310    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1f60d1-cf8f-4598-982d-9faea05bb01a-proxy-tls\") pod \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") "
May 20 15:44:35.568509 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.568352    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kserve-provision-location\") pod \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") "
May 20 15:44:35.568509 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.568409    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1f60d1-cf8f-4598-982d-9faea05bb01a-isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") pod \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\" (UID: \"bf1f60d1-cf8f-4598-982d-9faea05bb01a\") "
May 20 15:44:35.568753 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.568729    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf1f60d1-cf8f-4598-982d-9faea05bb01a" (UID: "bf1f60d1-cf8f-4598-982d-9faea05bb01a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:44:35.568824 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.568799    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1f60d1-cf8f-4598-982d-9faea05bb01a-isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config") pod "bf1f60d1-cf8f-4598-982d-9faea05bb01a" (UID: "bf1f60d1-cf8f-4598-982d-9faea05bb01a"). InnerVolumeSpecName "isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:44:35.570516 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.570485    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1f60d1-cf8f-4598-982d-9faea05bb01a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf1f60d1-cf8f-4598-982d-9faea05bb01a" (UID: "bf1f60d1-cf8f-4598-982d-9faea05bb01a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:44:35.570623 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.570522    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kube-api-access-p4r46" (OuterVolumeSpecName: "kube-api-access-p4r46") pod "bf1f60d1-cf8f-4598-982d-9faea05bb01a" (UID: "bf1f60d1-cf8f-4598-982d-9faea05bb01a"). InnerVolumeSpecName "kube-api-access-p4r46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:44:35.582201 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.582166    2570 generic.go:358] "Generic (PLEG): container finished" podID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerID="b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488" exitCode=0
May 20 15:44:35.582674 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.582248    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" event={"ID":"bf1f60d1-cf8f-4598-982d-9faea05bb01a","Type":"ContainerDied","Data":"b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488"}
May 20 15:44:35.582674 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.582265    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"
May 20 15:44:35.582674 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.582292    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s" event={"ID":"bf1f60d1-cf8f-4598-982d-9faea05bb01a","Type":"ContainerDied","Data":"eb0359b695e576189a98dc079198ce3d3b762490095d5dc938d65cdf73781f63"}
May 20 15:44:35.582674 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.582310    2570 scope.go:117] "RemoveContainer" containerID="8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9"
May 20 15:44:35.584238 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.584214    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz" event={"ID":"0b36bd82-8eeb-454e-9ef7-6d85bc801e0a","Type":"ContainerDied","Data":"835587e215034d3d277de0fc9e88cef5334d5471f4abb65821665a1d9df237a1"}
May 20 15:44:35.584346 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.584284    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"
May 20 15:44:35.591566 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.591434    2570 scope.go:117] "RemoveContainer" containerID="b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488"
May 20 15:44:35.600064 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.600042    2570 scope.go:117] "RemoveContainer" containerID="5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6"
May 20 15:44:35.608865 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.608843    2570 scope.go:117] "RemoveContainer" containerID="8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9"
May 20 15:44:35.609082 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.609060    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"]
May 20 15:44:35.609180 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:44:35.609164    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9\": container with ID starting with 8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9 not found: ID does not exist" containerID="8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9"
May 20 15:44:35.609221 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.609190    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9"} err="failed to get container status \"8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9\": rpc error: code = NotFound desc = could not find container \"8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9\": container with ID starting with 8b031a69adda7abb9ff1f305fb4092e99fe6c96c6cd907f0ec214283b4b2bda9 not found: ID does not exist"
May 20 15:44:35.609221 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.609208    2570 scope.go:117] "RemoveContainer" containerID="b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488"
May 20 15:44:35.609485 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:44:35.609466    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488\": container with ID starting with b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488 not found: ID does not exist" containerID="b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488"
May 20 15:44:35.609534 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.609494    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488"} err="failed to get container status \"b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488\": rpc error: code = NotFound desc = could not find container \"b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488\": container with ID starting with b5de8268761e6ee123b5d16bc6f3ec24c1eba67427081f4e0dc72761c1308488 not found: ID does not exist"
May 20 15:44:35.609534 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.609521    2570 scope.go:117] "RemoveContainer" containerID="5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6"
May 20 15:44:35.609775 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:44:35.609759    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6\": container with ID starting with 5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6 not found: ID does not exist" containerID="5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6"
May 20 15:44:35.609815 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.609781    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6"} err="failed to get container status \"5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6\": rpc error: code = NotFound desc = could not find container \"5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6\": container with ID starting with 5d2124bbde075f9cf381fb460c69e0e26378cb077ccdbab82f43d83d699e48a6 not found: ID does not exist"
May 20 15:44:35.609815 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.609797    2570 scope.go:117] "RemoveContainer" containerID="b7c9603fbdefadd38f3e5a1dfecb61f24c997fb2393701959f9297d4dedac288"
May 20 15:44:35.615509 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.615471    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-86fe3-predictor-8f4dc8849-j6z8s"]
May 20 15:44:35.619021 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.619003    2570 scope.go:117] "RemoveContainer" containerID="f87003359bcce86ce6fa490d09d8dffe764b924c4b2000a948a1b562d4f813f3"
May 20 15:44:35.625996 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.625967    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"]
May 20 15:44:35.627554 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.627540    2570 scope.go:117] "RemoveContainer" containerID="471909633eb241c42942bd843a0ffc46f595018f77857c56a687fbca3ef55e5a"
May 20 15:44:35.630672 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.630647    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-86fe3-predictor-786bd7b77c-ss9bz"]
May 20 15:44:35.670019 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.669987    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1f60d1-cf8f-4598-982d-9faea05bb01a-isvc-sklearn-graph-raw-hpa-86fe3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:44:35.670019 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.670016    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4r46\" (UniqueName: \"kubernetes.io/projected/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kube-api-access-p4r46\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:44:35.670212 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.670028    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1f60d1-cf8f-4598-982d-9faea05bb01a-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:44:35.670212 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.670041    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1f60d1-cf8f-4598-982d-9faea05bb01a-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:44:35.759321 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.759286    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" path="/var/lib/kubelet/pods/0b36bd82-8eeb-454e-9ef7-6d85bc801e0a/volumes"
May 20 15:44:35.759866 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:35.759851    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" path="/var/lib/kubelet/pods/bf1f60d1-cf8f-4598-982d-9faea05bb01a/volumes"
May 20 15:44:39.581302 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:39.581273    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:44:40.605310 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.605267    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"]
May 20 15:44:40.605922 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.605891    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container"
May 20 15:44:40.605922 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.605925    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.605948    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="storage-initializer"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.605957    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="storage-initializer"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.605966    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kube-rbac-proxy"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.605975    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kube-rbac-proxy"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606004    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606012    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606022    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kube-rbac-proxy"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606030    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kube-rbac-proxy"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606048    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="storage-initializer"
May 20 15:44:40.606112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606056    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="storage-initializer"
May 20 15:44:40.606541 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606155    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kserve-container"
May 20 15:44:40.606541 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606169    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kserve-container"
May 20 15:44:40.606541 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606182    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b36bd82-8eeb-454e-9ef7-6d85bc801e0a" containerName="kube-rbac-proxy"
May 20 15:44:40.606541 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.606190    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="bf1f60d1-cf8f-4598-982d-9faea05bb01a" containerName="kube-rbac-proxy"
May 20 15:44:40.609626 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.609601    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.612790 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.612772    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-47eff-predictor-serving-cert\""
May 20 15:44:40.613655 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.613636    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\""
May 20 15:44:40.613983 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.613956    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-proxy-tls\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.614112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.613995    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4rg\" (UniqueName: \"kubernetes.io/projected/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kube-api-access-7s4rg\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.614112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.614060    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kserve-provision-location\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.614112 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.614108    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.620274 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.620244    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"]
May 20 15:44:40.715062 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.715018    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-proxy-tls\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.715251 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.715080    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4rg\" (UniqueName: \"kubernetes.io/projected/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kube-api-access-7s4rg\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.715251 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.715118    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kserve-provision-location\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.715251 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.715184    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.715733 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.715704    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kserve-provision-location\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.716032 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.716011    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.717785 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.717761    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-proxy-tls\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.725825 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.725799    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4rg\" (UniqueName: \"kubernetes.io/projected/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kube-api-access-7s4rg\") pod \"isvc-logger-raw-47eff-predictor-5478d4946-jtzdg\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:40.921904 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:40.921805    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:41.061793 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:41.061718    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"]
May 20 15:44:41.064305 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:44:41.064274    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c06339d_3a70_49f8_8ecf_70aa9c8c30d6.slice/crio-02d29dedf2e850e63c6afe71b3bea70304af56856ab7c8563f7220a3f42f315b WatchSource:0}: Error finding container 02d29dedf2e850e63c6afe71b3bea70304af56856ab7c8563f7220a3f42f315b: Status 404 returned error can't find the container with id 02d29dedf2e850e63c6afe71b3bea70304af56856ab7c8563f7220a3f42f315b
May 20 15:44:41.614845 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:41.614805    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerStarted","Data":"5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc"}
May 20 15:44:41.614845 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:41.614845    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerStarted","Data":"02d29dedf2e850e63c6afe71b3bea70304af56856ab7c8563f7220a3f42f315b"}
May 20 15:44:45.628704 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:45.628668    2570 generic.go:358] "Generic (PLEG): container finished" podID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerID="5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc" exitCode=0
May 20 15:44:45.629038 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:45.628741    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerDied","Data":"5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc"}
May 20 15:44:46.634187 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:46.634144    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerStarted","Data":"c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826"}
May 20 15:44:46.634187 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:46.634186    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerStarted","Data":"4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6"}
May 20 15:44:46.634187 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:46.634197    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerStarted","Data":"e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9"}
May 20 15:44:46.634661 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:46.634546    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:46.634704 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:46.634676    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:46.636054 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:46.636030    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:44:46.658615 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:46.658552    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podStartSLOduration=6.6585363090000005 podStartE2EDuration="6.658536309s" podCreationTimestamp="2026-05-20 15:44:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:44:46.655864698 +0000 UTC m=+775.485960620" watchObservedRunningTime="2026-05-20 15:44:46.658536309 +0000 UTC m=+775.488632230"
May 20 15:44:47.638133 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:47.638101    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:47.638525 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:47.638240    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:44:47.639329 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:47.639300    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:48.641445 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:48.641396    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:44:48.641969 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:48.641943    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:53.646334 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:53.646301    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:44:53.646870 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:53.646840    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:44:53.647189 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:44:53.647156    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:03.646879 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:03.646834    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:45:03.647306 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:03.647208    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:13.647091 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:13.646992    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:45:13.647616 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:13.647590    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:23.647118 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:23.647062    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:45:23.647639 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:23.647618    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:33.647160 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:33.647107    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:45:33.647627 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:33.647505    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:43.647472 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:43.647414    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:45:43.647872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:43.647845    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:53.647750 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:53.647715    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:45:53.648200 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:45:53.647922    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:46:05.569899 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.569865    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc_bc63f01a-f82a-4ef8-b174-83adb94e1fad/kserve-container/0.log"
May 20 15:46:05.751171 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.751132    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"]
May 20 15:46:05.751704 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.751630    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" containerID="cri-o://e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9" gracePeriod=30
May 20 15:46:05.751704 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.751660    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy" containerID="cri-o://4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6" gracePeriod=30
May 20 15:46:05.751887 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.751668    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" containerID="cri-o://c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826" gracePeriod=30
May 20 15:46:05.803819 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.803775    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"]
May 20 15:46:05.808098 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.808071    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:05.810708 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.810682    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\""
May 20 15:46:05.810867 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.810680    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-db50d-predictor-serving-cert\""
May 20 15:46:05.816521 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.816349    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"]
May 20 15:46:05.869033 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.868940    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"]
May 20 15:46:05.869267 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.869235    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc" podUID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerName="kserve-container" containerID="cri-o://b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7" gracePeriod=30
May 20 15:46:05.869267 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.869247    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc" podUID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerName="kube-rbac-proxy" containerID="cri-o://48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60" gracePeriod=30
May 20 15:46:05.903086 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.903037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902200ca-05c3-417b-85ce-5799d3e757f4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:05.903267 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.903163    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902200ca-05c3-417b-85ce-5799d3e757f4-proxy-tls\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:05.903267 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.903207    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbtj2\" (UniqueName: \"kubernetes.io/projected/902200ca-05c3-417b-85ce-5799d3e757f4-kube-api-access-mbtj2\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:05.903353 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.903297    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902200ca-05c3-417b-85ce-5799d3e757f4-isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:05.908281 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.908252    2570 generic.go:358] "Generic (PLEG): container finished" podID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerID="4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6" exitCode=2
May 20 15:46:05.908455 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:05.908309    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerDied","Data":"4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6"}
May 20 15:46:06.004528 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.004483    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902200ca-05c3-417b-85ce-5799d3e757f4-isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:06.004728 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.004540    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902200ca-05c3-417b-85ce-5799d3e757f4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:06.004728 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.004608    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902200ca-05c3-417b-85ce-5799d3e757f4-proxy-tls\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:06.004728 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.004634    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mbtj2\" (UniqueName: \"kubernetes.io/projected/902200ca-05c3-417b-85ce-5799d3e757f4-kube-api-access-mbtj2\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:06.005032 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.005006    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902200ca-05c3-417b-85ce-5799d3e757f4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:06.005248 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.005219    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902200ca-05c3-417b-85ce-5799d3e757f4-isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:06.007189 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.007161    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902200ca-05c3-417b-85ce-5799d3e757f4-proxy-tls\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:06.013719 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.013684    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbtj2\" (UniqueName: \"kubernetes.io/projected/902200ca-05c3-417b-85ce-5799d3e757f4-kube-api-access-mbtj2\") pod \"isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:06.110307 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.110280    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:46:06.121499 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.121417    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:06.207226 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.206892    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc63f01a-f82a-4ef8-b174-83adb94e1fad-proxy-tls\") pod \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") "
May 20 15:46:06.207226 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.206950    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qxrh\" (UniqueName: \"kubernetes.io/projected/bc63f01a-f82a-4ef8-b174-83adb94e1fad-kube-api-access-9qxrh\") pod \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") "
May 20 15:46:06.207226 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.207041    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc63f01a-f82a-4ef8-b174-83adb94e1fad-message-dumper-raw-47eff-kube-rbac-proxy-sar-config\") pod \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\" (UID: \"bc63f01a-f82a-4ef8-b174-83adb94e1fad\") "
May 20 15:46:06.207614 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.207572    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc63f01a-f82a-4ef8-b174-83adb94e1fad-message-dumper-raw-47eff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-raw-47eff-kube-rbac-proxy-sar-config") pod "bc63f01a-f82a-4ef8-b174-83adb94e1fad" (UID: "bc63f01a-f82a-4ef8-b174-83adb94e1fad"). InnerVolumeSpecName "message-dumper-raw-47eff-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:46:06.211872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.211843    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc63f01a-f82a-4ef8-b174-83adb94e1fad-kube-api-access-9qxrh" (OuterVolumeSpecName: "kube-api-access-9qxrh") pod "bc63f01a-f82a-4ef8-b174-83adb94e1fad" (UID: "bc63f01a-f82a-4ef8-b174-83adb94e1fad"). InnerVolumeSpecName "kube-api-access-9qxrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:46:06.214681 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.214649    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc63f01a-f82a-4ef8-b174-83adb94e1fad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc63f01a-f82a-4ef8-b174-83adb94e1fad" (UID: "bc63f01a-f82a-4ef8-b174-83adb94e1fad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:46:06.259776 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.259746    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"]
May 20 15:46:06.262394 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:46:06.262352    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902200ca_05c3_417b_85ce_5799d3e757f4.slice/crio-d6c3c923470dd4d4b8dff19244f4deda811f5e3b68a107aff815462ba288cdda WatchSource:0}: Error finding container d6c3c923470dd4d4b8dff19244f4deda811f5e3b68a107aff815462ba288cdda: Status 404 returned error can't find the container with id d6c3c923470dd4d4b8dff19244f4deda811f5e3b68a107aff815462ba288cdda
May 20 15:46:06.308124 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.308093    2570 reconciler_common.go:299] "Volume detached for volume \"message-dumper-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc63f01a-f82a-4ef8-b174-83adb94e1fad-message-dumper-raw-47eff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:46:06.308124 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.308122    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc63f01a-f82a-4ef8-b174-83adb94e1fad-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:46:06.308326 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.308133    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qxrh\" (UniqueName: \"kubernetes.io/projected/bc63f01a-f82a-4ef8-b174-83adb94e1fad-kube-api-access-9qxrh\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:46:06.916256 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.916218    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" event={"ID":"902200ca-05c3-417b-85ce-5799d3e757f4","Type":"ContainerStarted","Data":"826a12b7186afa244d87f61ab7a16aa8e039c2103cc047d09fad316ebc05d3a0"}
May 20 15:46:06.916256 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.916262    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" event={"ID":"902200ca-05c3-417b-85ce-5799d3e757f4","Type":"ContainerStarted","Data":"d6c3c923470dd4d4b8dff19244f4deda811f5e3b68a107aff815462ba288cdda"}
May 20 15:46:06.917712 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.917685    2570 generic.go:358] "Generic (PLEG): container finished" podID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerID="48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60" exitCode=2
May 20 15:46:06.917712 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.917706    2570 generic.go:358] "Generic (PLEG): container finished" podID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerID="b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7" exitCode=2
May 20 15:46:06.917878 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.917733    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc" event={"ID":"bc63f01a-f82a-4ef8-b174-83adb94e1fad","Type":"ContainerDied","Data":"48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60"}
May 20 15:46:06.917878 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.917747    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"
May 20 15:46:06.917878 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.917762    2570 scope.go:117] "RemoveContainer" containerID="48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60"
May 20 15:46:06.917878 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.917752    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc" event={"ID":"bc63f01a-f82a-4ef8-b174-83adb94e1fad","Type":"ContainerDied","Data":"b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7"}
May 20 15:46:06.917878 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.917862    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc" event={"ID":"bc63f01a-f82a-4ef8-b174-83adb94e1fad","Type":"ContainerDied","Data":"6a5d4cef6048c0f4eb705a11c79f356082e6bb98ab8c76bb43c422a871dc2b02"}
May 20 15:46:06.926655 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.926636    2570 scope.go:117] "RemoveContainer" containerID="b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7"
May 20 15:46:06.936252 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.936227    2570 scope.go:117] "RemoveContainer" containerID="48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60"
May 20 15:46:06.944879 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:46:06.944843    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60\": container with ID starting with 48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60 not found: ID does not exist" containerID="48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60"
May 20 15:46:06.945043 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.944885    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60"} err="failed to get container status \"48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60\": rpc error: code = NotFound desc = could not find container \"48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60\": container with ID starting with 48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60 not found: ID does not exist"
May 20 15:46:06.945043 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.944908    2570 scope.go:117] "RemoveContainer" containerID="b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7"
May 20 15:46:06.945275 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:46:06.945242    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7\": container with ID starting with b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7 not found: ID does not exist" containerID="b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7"
May 20 15:46:06.945312 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.945289    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7"} err="failed to get container status \"b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7\": rpc error: code = NotFound desc = could not find container \"b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7\": container with ID starting with b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7 not found: ID does not exist"
May 20 15:46:06.945358 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.945317    2570 scope.go:117] "RemoveContainer" containerID="48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60"
May 20 15:46:06.945653 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.945627    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60"} err="failed to get container status \"48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60\": rpc error: code = NotFound desc = could not find container \"48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60\": container with ID starting with 48d11df6f47ce116fbcc8f91518b22cef10e9a064131f905148aa86b0e1c4e60 not found: ID does not exist"
May 20 15:46:06.945725 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.945655    2570 scope.go:117] "RemoveContainer" containerID="b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7"
May 20 15:46:06.945919 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.945898    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7"} err="failed to get container status \"b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7\": rpc error: code = NotFound desc = could not find container \"b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7\": container with ID starting with b904a02860dc376ed1c4305e934b69d09c6d4064987e114a88739cfc161a4db7 not found: ID does not exist"
May 20 15:46:06.951721 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.951684    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"]
May 20 15:46:06.953630 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:06.953603    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-47eff-predictor-98bc6cb9d-57wjc"]
May 20 15:46:07.759880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:07.759841    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" path="/var/lib/kubelet/pods/bc63f01a-f82a-4ef8-b174-83adb94e1fad/volumes"
May 20 15:46:08.642097 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:08.642054    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 15:46:10.932619 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:10.932587    2570 generic.go:358] "Generic (PLEG): container finished" podID="902200ca-05c3-417b-85ce-5799d3e757f4" containerID="826a12b7186afa244d87f61ab7a16aa8e039c2103cc047d09fad316ebc05d3a0" exitCode=0
May 20 15:46:10.932997 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:10.932662    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" event={"ID":"902200ca-05c3-417b-85ce-5799d3e757f4","Type":"ContainerDied","Data":"826a12b7186afa244d87f61ab7a16aa8e039c2103cc047d09fad316ebc05d3a0"}
May 20 15:46:10.934800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:10.934773    2570 generic.go:358] "Generic (PLEG): container finished" podID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerID="e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9" exitCode=0
May 20 15:46:10.934907 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:10.934828    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerDied","Data":"e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9"}
May 20 15:46:11.939461 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:11.939431    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" event={"ID":"902200ca-05c3-417b-85ce-5799d3e757f4","Type":"ContainerStarted","Data":"24006caf2748a00a9878a7ae6ffece485198e3138e5feb27bf88c0fdf2e89546"}
May 20 15:46:11.939837 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:11.939470    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" event={"ID":"902200ca-05c3-417b-85ce-5799d3e757f4","Type":"ContainerStarted","Data":"31337acc40b4213285f015dc75717f7041cf7dc926b2844253d5e5f8789d0db5"}
May 20 15:46:11.939837 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:11.939668    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:11.962579 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:11.962530    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podStartSLOduration=6.96251765 podStartE2EDuration="6.96251765s" podCreationTimestamp="2026-05-20 15:46:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:46:11.961304225 +0000 UTC m=+860.791400147" watchObservedRunningTime="2026-05-20 15:46:11.96251765 +0000 UTC m=+860.792613571"
May 20 15:46:12.943122 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:12.943085    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:12.944430 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:12.944395    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:46:13.642037 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:13.641994    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 15:46:13.647566 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:13.647523    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:46:13.647864 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:13.647836    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:13.947082 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:13.946977    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:46:18.641889 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:18.641844    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 15:46:18.642295 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:18.641983    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:46:18.952145 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:18.952059    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:46:18.952722 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:18.952689    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:46:23.642329 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:23.642271    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 15:46:23.647873 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:23.647829    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:46:23.648207 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:23.648187    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:28.642030 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:28.641979    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 15:46:28.952843 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:28.952794    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:46:33.641743 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:33.641686    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 15:46:33.647119 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:33.647075    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 15:46:33.647256 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:33.647211    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:46:33.647438 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:33.647414    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:33.647529 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:33.647518    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:46:35.909616 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:35.909598    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:46:36.029076 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.028995    2570 generic.go:358] "Generic (PLEG): container finished" podID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerID="c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826" exitCode=0
May 20 15:46:36.029208 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.029068    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerDied","Data":"c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826"}
May 20 15:46:36.029208 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.029095    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"
May 20 15:46:36.029208 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.029107    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg" event={"ID":"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6","Type":"ContainerDied","Data":"02d29dedf2e850e63c6afe71b3bea70304af56856ab7c8563f7220a3f42f315b"}
May 20 15:46:36.029208 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.029123    2570 scope.go:117] "RemoveContainer" containerID="c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826"
May 20 15:46:36.036870 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.036854    2570 scope.go:117] "RemoveContainer" containerID="4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6"
May 20 15:46:36.044534 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.044518    2570 scope.go:117] "RemoveContainer" containerID="e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9"
May 20 15:46:36.051559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.051543    2570 scope.go:117] "RemoveContainer" containerID="5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc"
May 20 15:46:36.064280 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.064262    2570 scope.go:117] "RemoveContainer" containerID="c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826"
May 20 15:46:36.064619 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:46:36.064600    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826\": container with ID starting with c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826 not found: ID does not exist" containerID="c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826"
May 20 15:46:36.064687 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.064626    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826"} err="failed to get container status \"c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826\": rpc error: code = NotFound desc = could not find container \"c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826\": container with ID starting with c387145c0a077dbbf97356e5e9573738de9028172b4f32a7affb3b1ba6a29826 not found: ID does not exist"
May 20 15:46:36.064687 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.064645    2570 scope.go:117] "RemoveContainer" containerID="4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6"
May 20 15:46:36.064868 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:46:36.064851    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6\": container with ID starting with 4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6 not found: ID does not exist" containerID="4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6"
May 20 15:46:36.064910 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.064875    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6"} err="failed to get container status \"4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6\": rpc error: code = NotFound desc = could not find container \"4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6\": container with ID starting with 4f1213a745857b495362aef71022ed47067843000ab76efd6bcbb8bef9e3ade6 not found: ID does not exist"
May 20 15:46:36.064910 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.064890    2570 scope.go:117] "RemoveContainer" containerID="e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9"
May 20 15:46:36.065094 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:46:36.065081    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9\": container with ID starting with e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9 not found: ID does not exist" containerID="e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9"
May 20 15:46:36.065133 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.065096    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9"} err="failed to get container status \"e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9\": rpc error: code = NotFound desc = could not find container \"e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9\": container with ID starting with e240777092c68b148001c542cd797449911051b6a64fa49593185c6e611745f9 not found: ID does not exist"
May 20 15:46:36.065133 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.065107    2570 scope.go:117] "RemoveContainer" containerID="5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc"
May 20 15:46:36.065302 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:46:36.065286    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc\": container with ID starting with 5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc not found: ID does not exist" containerID="5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc"
May 20 15:46:36.065352 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.065302    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc"} err="failed to get container status \"5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc\": rpc error: code = NotFound desc = could not find container \"5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc\": container with ID starting with 5a081f48b8a7871ce9d014fe27338c2bf16397cc63d07a60a6c8dcf5617fcfbc not found: ID does not exist"
May 20 15:46:36.097798 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.097774    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\") pod \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") "
May 20 15:46:36.097900 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.097812    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-proxy-tls\") pod \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") "
May 20 15:46:36.097900 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.097852    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kserve-provision-location\") pod \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") "
May 20 15:46:36.098008 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.097903    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s4rg\" (UniqueName: \"kubernetes.io/projected/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kube-api-access-7s4rg\") pod \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\" (UID: \"1c06339d-3a70-49f8-8ecf-70aa9c8c30d6\") "
May 20 15:46:36.098168 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.098146    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-isvc-logger-raw-47eff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-raw-47eff-kube-rbac-proxy-sar-config") pod "1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" (UID: "1c06339d-3a70-49f8-8ecf-70aa9c8c30d6"). InnerVolumeSpecName "isvc-logger-raw-47eff-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:46:36.098224 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.098192    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" (UID: "1c06339d-3a70-49f8-8ecf-70aa9c8c30d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:46:36.099843 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.099821    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" (UID: "1c06339d-3a70-49f8-8ecf-70aa9c8c30d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:46:36.099915 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.099868    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kube-api-access-7s4rg" (OuterVolumeSpecName: "kube-api-access-7s4rg") pod "1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" (UID: "1c06339d-3a70-49f8-8ecf-70aa9c8c30d6"). InnerVolumeSpecName "kube-api-access-7s4rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:46:36.198547 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.198507    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-isvc-logger-raw-47eff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:46:36.198547 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.198546    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:46:36.198547 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.198559    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:46:36.198743 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.198570    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s4rg\" (UniqueName: \"kubernetes.io/projected/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6-kube-api-access-7s4rg\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:46:36.353657 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.353630    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"]
May 20 15:46:36.357415 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:36.357392    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-47eff-predictor-5478d4946-jtzdg"]
May 20 15:46:37.759397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:37.759336    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" path="/var/lib/kubelet/pods/1c06339d-3a70-49f8-8ecf-70aa9c8c30d6/volumes"
May 20 15:46:38.953146 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:38.953102    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:46:48.953220 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:48.953175    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:46:51.690724 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:51.690698    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:46:51.692945 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:51.692922    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:46:58.953561 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:46:58.953519    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:47:08.953494 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:47:08.953455    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:47:18.953026 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:47:18.952985    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:47:23.756319 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:47:23.756268    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:47:33.756711 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:47:33.756661    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:47:43.756532 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:47:43.756488    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:47:53.756873 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:47:53.756824    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:48:03.756649 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:03.756601    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:48:13.756951 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:13.756846    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:48:23.756969 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:23.756922    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:48:33.759515 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:33.759484    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:48:35.991805 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:35.991767    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"]
May 20 15:48:35.992229 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:35.992054    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" containerID="cri-o://31337acc40b4213285f015dc75717f7041cf7dc926b2844253d5e5f8789d0db5" gracePeriod=30
May 20 15:48:35.992229 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:35.992115    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kube-rbac-proxy" containerID="cri-o://24006caf2748a00a9878a7ae6ffece485198e3138e5feb27bf88c0fdf2e89546" gracePeriod=30
May 20 15:48:36.144303 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144263    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"]
May 20 15:48:36.144666 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144652    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="storage-initializer"
May 20 15:48:36.144726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144667    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="storage-initializer"
May 20 15:48:36.144726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144681    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container"
May 20 15:48:36.144726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144687    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container"
May 20 15:48:36.144726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144698    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerName="kserve-container"
May 20 15:48:36.144726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144703    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerName="kserve-container"
May 20 15:48:36.144726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144715    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent"
May 20 15:48:36.144726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144721    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent"
May 20 15:48:36.144726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144728    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerName="kube-rbac-proxy"
May 20 15:48:36.144992 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144733    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerName="kube-rbac-proxy"
May 20 15:48:36.144992 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144740    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy"
May 20 15:48:36.144992 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144746    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy"
May 20 15:48:36.144992 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144803    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="agent"
May 20 15:48:36.144992 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144813    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerName="kube-rbac-proxy"
May 20 15:48:36.144992 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144820    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kube-rbac-proxy"
May 20 15:48:36.144992 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144827    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c06339d-3a70-49f8-8ecf-70aa9c8c30d6" containerName="kserve-container"
May 20 15:48:36.144992 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.144834    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc63f01a-f82a-4ef8-b174-83adb94e1fad" containerName="kserve-container"
May 20 15:48:36.148144 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.148124    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.150508 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.150485    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c3463e-predictor-serving-cert\""
May 20 15:48:36.150734 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.150717    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c3463e-kube-rbac-proxy-sar-config\""
May 20 15:48:36.159730 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.159702    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"]
May 20 15:48:36.268090 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.267989    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3425f3d9-1734-4ad9-bec4-d94e85c206ec-proxy-tls\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.268090 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.268052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm982\" (UniqueName: \"kubernetes.io/projected/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kube-api-access-qm982\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.268273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.268095    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kserve-provision-location\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.268273 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.268131    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3425f3d9-1734-4ad9-bec4-d94e85c206ec-isvc-primary-c3463e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.369615 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.369567    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qm982\" (UniqueName: \"kubernetes.io/projected/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kube-api-access-qm982\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.369828 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.369635    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kserve-provision-location\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.369828 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.369688    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3425f3d9-1734-4ad9-bec4-d94e85c206ec-isvc-primary-c3463e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.369828 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.369752    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3425f3d9-1734-4ad9-bec4-d94e85c206ec-proxy-tls\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.370057 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.369996    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kserve-provision-location\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.370330 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.370307    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3425f3d9-1734-4ad9-bec4-d94e85c206ec-isvc-primary-c3463e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.372351 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.372324    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3425f3d9-1734-4ad9-bec4-d94e85c206ec-proxy-tls\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.379879 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.379850    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm982\" (UniqueName: \"kubernetes.io/projected/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kube-api-access-qm982\") pod \"isvc-primary-c3463e-predictor-86cdb4986d-qqmd4\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") " pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.435087 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.435053    2570 generic.go:358] "Generic (PLEG): container finished" podID="902200ca-05c3-417b-85ce-5799d3e757f4" containerID="24006caf2748a00a9878a7ae6ffece485198e3138e5feb27bf88c0fdf2e89546" exitCode=2
May 20 15:48:36.435234 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.435128    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" event={"ID":"902200ca-05c3-417b-85ce-5799d3e757f4","Type":"ContainerDied","Data":"24006caf2748a00a9878a7ae6ffece485198e3138e5feb27bf88c0fdf2e89546"}
May 20 15:48:36.459396 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.459325    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:36.604208 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:36.604180    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"]
May 20 15:48:36.606914 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:48:36.606875    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3425f3d9_1734_4ad9_bec4_d94e85c206ec.slice/crio-0cbfe5e7ce02b269e1c554720e0c02357a820a56e5d395cf8fb334c98d066c0f WatchSource:0}: Error finding container 0cbfe5e7ce02b269e1c554720e0c02357a820a56e5d395cf8fb334c98d066c0f: Status 404 returned error can't find the container with id 0cbfe5e7ce02b269e1c554720e0c02357a820a56e5d395cf8fb334c98d066c0f
May 20 15:48:37.439604 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:37.439565    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" event={"ID":"3425f3d9-1734-4ad9-bec4-d94e85c206ec","Type":"ContainerStarted","Data":"f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959"}
May 20 15:48:37.439604 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:37.439601    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" event={"ID":"3425f3d9-1734-4ad9-bec4-d94e85c206ec","Type":"ContainerStarted","Data":"0cbfe5e7ce02b269e1c554720e0c02357a820a56e5d395cf8fb334c98d066c0f"}
May 20 15:48:38.947710 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:38.947659    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
May 20 15:48:41.454776 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:41.454741    2570 generic.go:358] "Generic (PLEG): container finished" podID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerID="f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959" exitCode=0
May 20 15:48:41.455182 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:41.454810    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" event={"ID":"3425f3d9-1734-4ad9-bec4-d94e85c206ec","Type":"ContainerDied","Data":"f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959"}
May 20 15:48:42.459652 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:42.459612    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" event={"ID":"3425f3d9-1734-4ad9-bec4-d94e85c206ec","Type":"ContainerStarted","Data":"52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163"}
May 20 15:48:42.460136 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:42.459660    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" event={"ID":"3425f3d9-1734-4ad9-bec4-d94e85c206ec","Type":"ContainerStarted","Data":"933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0"}
May 20 15:48:42.460136 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:42.459920    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:43.463637 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:43.463601    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:43.464988 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:43.464957    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 15:48:43.756868 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:43.756771    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 15:48:43.948212 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:43.948168    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
May 20 15:48:44.467173 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:44.467130    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 15:48:45.473402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.473351    2570 generic.go:358] "Generic (PLEG): container finished" podID="902200ca-05c3-417b-85ce-5799d3e757f4" containerID="31337acc40b4213285f015dc75717f7041cf7dc926b2844253d5e5f8789d0db5" exitCode=0
May 20 15:48:45.473402 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.473401    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" event={"ID":"902200ca-05c3-417b-85ce-5799d3e757f4","Type":"ContainerDied","Data":"31337acc40b4213285f015dc75717f7041cf7dc926b2844253d5e5f8789d0db5"}
May 20 15:48:45.541391 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.541339    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:48:45.561399 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.561330    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podStartSLOduration=9.561308755 podStartE2EDuration="9.561308755s" podCreationTimestamp="2026-05-20 15:48:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:48:42.479400141 +0000 UTC m=+1011.309496059" watchObservedRunningTime="2026-05-20 15:48:45.561308755 +0000 UTC m=+1014.391404675"
May 20 15:48:45.668085 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.667985    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902200ca-05c3-417b-85ce-5799d3e757f4-proxy-tls\") pod \"902200ca-05c3-417b-85ce-5799d3e757f4\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") "
May 20 15:48:45.668085 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.668036    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902200ca-05c3-417b-85ce-5799d3e757f4-isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\") pod \"902200ca-05c3-417b-85ce-5799d3e757f4\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") "
May 20 15:48:45.668085 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.668073    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902200ca-05c3-417b-85ce-5799d3e757f4-kserve-provision-location\") pod \"902200ca-05c3-417b-85ce-5799d3e757f4\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") "
May 20 15:48:45.668395 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.668138    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbtj2\" (UniqueName: \"kubernetes.io/projected/902200ca-05c3-417b-85ce-5799d3e757f4-kube-api-access-mbtj2\") pod \"902200ca-05c3-417b-85ce-5799d3e757f4\" (UID: \"902200ca-05c3-417b-85ce-5799d3e757f4\") "
May 20 15:48:45.668487 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.668459    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902200ca-05c3-417b-85ce-5799d3e757f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "902200ca-05c3-417b-85ce-5799d3e757f4" (UID: "902200ca-05c3-417b-85ce-5799d3e757f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:48:45.668528 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.668457    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902200ca-05c3-417b-85ce-5799d3e757f4-isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config") pod "902200ca-05c3-417b-85ce-5799d3e757f4" (UID: "902200ca-05c3-417b-85ce-5799d3e757f4"). InnerVolumeSpecName "isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:48:45.670294 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.670271    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902200ca-05c3-417b-85ce-5799d3e757f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "902200ca-05c3-417b-85ce-5799d3e757f4" (UID: "902200ca-05c3-417b-85ce-5799d3e757f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:48:45.670354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.670320    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902200ca-05c3-417b-85ce-5799d3e757f4-kube-api-access-mbtj2" (OuterVolumeSpecName: "kube-api-access-mbtj2") pod "902200ca-05c3-417b-85ce-5799d3e757f4" (UID: "902200ca-05c3-417b-85ce-5799d3e757f4"). InnerVolumeSpecName "kube-api-access-mbtj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:48:45.769646 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.769617    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbtj2\" (UniqueName: \"kubernetes.io/projected/902200ca-05c3-417b-85ce-5799d3e757f4-kube-api-access-mbtj2\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:48:45.769646 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.769647    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902200ca-05c3-417b-85ce-5799d3e757f4-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:48:45.769820 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.769662    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902200ca-05c3-417b-85ce-5799d3e757f4-isvc-sklearn-scale-raw-db50d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:48:45.769820 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:45.769676    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902200ca-05c3-417b-85ce-5799d3e757f4-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:48:46.478249 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:46.478211    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz" event={"ID":"902200ca-05c3-417b-85ce-5799d3e757f4","Type":"ContainerDied","Data":"d6c3c923470dd4d4b8dff19244f4deda811f5e3b68a107aff815462ba288cdda"}
May 20 15:48:46.478726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:46.478259    2570 scope.go:117] "RemoveContainer" containerID="24006caf2748a00a9878a7ae6ffece485198e3138e5feb27bf88c0fdf2e89546"
May 20 15:48:46.478726 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:46.478221    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"
May 20 15:48:46.486965 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:46.486944    2570 scope.go:117] "RemoveContainer" containerID="31337acc40b4213285f015dc75717f7041cf7dc926b2844253d5e5f8789d0db5"
May 20 15:48:46.494604 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:46.494583    2570 scope.go:117] "RemoveContainer" containerID="826a12b7186afa244d87f61ab7a16aa8e039c2103cc047d09fad316ebc05d3a0"
May 20 15:48:46.497721 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:46.497692    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"]
May 20 15:48:46.503229 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:46.503198    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-db50d-predictor-8f6c55c9-pltfz"]
May 20 15:48:47.760101 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:47.760055    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" path="/var/lib/kubelet/pods/902200ca-05c3-417b-85ce-5799d3e757f4/volumes"
May 20 15:48:49.472047 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:49.472017    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:48:49.472643 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:49.472617    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 15:48:59.473594 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:48:59.473549    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 15:49:09.472815 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:09.472771    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 15:49:19.472626 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:19.472581    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 15:49:29.473411 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:29.473354    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 15:49:39.473488 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:39.473405    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 15:49:49.473593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:49.473559    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:49:56.234187 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.234154    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"]
May 20 15:49:56.234675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.234554    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="storage-initializer"
May 20 15:49:56.234675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.234566    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="storage-initializer"
May 20 15:49:56.234675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.234573    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kube-rbac-proxy"
May 20 15:49:56.234675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.234579    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kube-rbac-proxy"
May 20 15:49:56.234675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.234587    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container"
May 20 15:49:56.234675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.234592    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container"
May 20 15:49:56.234675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.234651    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kube-rbac-proxy"
May 20 15:49:56.234675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.234662    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="902200ca-05c3-417b-85ce-5799d3e757f4" containerName="kserve-container"
May 20 15:49:56.238183 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.238164    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.240699 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.240675    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c3463e-predictor-serving-cert\""
May 20 15:49:56.240699 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.240687    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-c3463e-dockercfg-6h6r2\""
May 20 15:49:56.240843 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.240730    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-c3463e\""
May 20 15:49:56.240843 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.240730    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c3463e-kube-rbac-proxy-sar-config\""
May 20 15:49:56.240843 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.240795    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 15:49:56.247946 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.247924    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"]
May 20 15:49:56.284911 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.284886    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/882f2be3-ade1-4f59-a2db-a37617e8279d-kserve-provision-location\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.285041 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.284924    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/882f2be3-ade1-4f59-a2db-a37617e8279d-proxy-tls\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.285041 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.284943    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlsn6\" (UniqueName: \"kubernetes.io/projected/882f2be3-ade1-4f59-a2db-a37617e8279d-kube-api-access-dlsn6\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.285133 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.285054    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-cabundle-cert\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.285133 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.285093    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-isvc-secondary-c3463e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.385740 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.385694    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-isvc-secondary-c3463e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.385945 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.385761    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/882f2be3-ade1-4f59-a2db-a37617e8279d-kserve-provision-location\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.385945 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.385807    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/882f2be3-ade1-4f59-a2db-a37617e8279d-proxy-tls\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.385945 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.385835    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dlsn6\" (UniqueName: \"kubernetes.io/projected/882f2be3-ade1-4f59-a2db-a37617e8279d-kube-api-access-dlsn6\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.385945 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.385925    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-cabundle-cert\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.386184 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:49:56.385973    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-serving-cert: secret "isvc-secondary-c3463e-predictor-serving-cert" not found
May 20 15:49:56.386184 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:49:56.386063    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/882f2be3-ade1-4f59-a2db-a37617e8279d-proxy-tls podName:882f2be3-ade1-4f59-a2db-a37617e8279d nodeName:}" failed. No retries permitted until 2026-05-20 15:49:56.886041889 +0000 UTC m=+1085.716137797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/882f2be3-ade1-4f59-a2db-a37617e8279d-proxy-tls") pod "isvc-secondary-c3463e-predictor-746949d687-t525f" (UID: "882f2be3-ade1-4f59-a2db-a37617e8279d") : secret "isvc-secondary-c3463e-predictor-serving-cert" not found
May 20 15:49:56.386184 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.386131    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/882f2be3-ade1-4f59-a2db-a37617e8279d-kserve-provision-location\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.386475 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.386454    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-isvc-secondary-c3463e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.386568 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.386555    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-cabundle-cert\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.394761 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.394730    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlsn6\" (UniqueName: \"kubernetes.io/projected/882f2be3-ade1-4f59-a2db-a37617e8279d-kube-api-access-dlsn6\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.890764 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.890721    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/882f2be3-ade1-4f59-a2db-a37617e8279d-proxy-tls\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:56.893333 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:56.893296    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/882f2be3-ade1-4f59-a2db-a37617e8279d-proxy-tls\") pod \"isvc-secondary-c3463e-predictor-746949d687-t525f\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") " pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:57.149282 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:57.149179    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:49:57.296294 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:57.296216    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"]
May 20 15:49:57.301989 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:49:57.301917    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882f2be3_ade1_4f59_a2db_a37617e8279d.slice/crio-09961e7c507a0aca1601558bab0c76da070ec3ad578536f9acd08e1fa82e4139 WatchSource:0}: Error finding container 09961e7c507a0aca1601558bab0c76da070ec3ad578536f9acd08e1fa82e4139: Status 404 returned error can't find the container with id 09961e7c507a0aca1601558bab0c76da070ec3ad578536f9acd08e1fa82e4139
May 20 15:49:57.303436 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:57.303416    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:49:57.716676 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:57.716640    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f" event={"ID":"882f2be3-ade1-4f59-a2db-a37617e8279d","Type":"ContainerStarted","Data":"c7203e5cac2aba7585437e470b077a6996901f6a19b4e3371d46d483e6ee2636"}
May 20 15:49:57.716676 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:49:57.716682    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f" event={"ID":"882f2be3-ade1-4f59-a2db-a37617e8279d","Type":"ContainerStarted","Data":"09961e7c507a0aca1601558bab0c76da070ec3ad578536f9acd08e1fa82e4139"}
May 20 15:50:03.738414 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:03.738360    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c3463e-predictor-746949d687-t525f_882f2be3-ade1-4f59-a2db-a37617e8279d/storage-initializer/0.log"
May 20 15:50:03.738810 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:03.738422    2570 generic.go:358] "Generic (PLEG): container finished" podID="882f2be3-ade1-4f59-a2db-a37617e8279d" containerID="c7203e5cac2aba7585437e470b077a6996901f6a19b4e3371d46d483e6ee2636" exitCode=1
May 20 15:50:03.738810 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:03.738457    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f" event={"ID":"882f2be3-ade1-4f59-a2db-a37617e8279d","Type":"ContainerDied","Data":"c7203e5cac2aba7585437e470b077a6996901f6a19b4e3371d46d483e6ee2636"}
May 20 15:50:04.746200 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:04.746171    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c3463e-predictor-746949d687-t525f_882f2be3-ade1-4f59-a2db-a37617e8279d/storage-initializer/0.log"
May 20 15:50:04.746627 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:04.746276    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f" event={"ID":"882f2be3-ade1-4f59-a2db-a37617e8279d","Type":"ContainerStarted","Data":"092aaa38c3890651cecac95d947b92e34a410439dfeb13f87d4ef939460a7a1a"}
May 20 15:50:08.761786 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:08.761757    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c3463e-predictor-746949d687-t525f_882f2be3-ade1-4f59-a2db-a37617e8279d/storage-initializer/1.log"
May 20 15:50:08.762195 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:08.762143    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c3463e-predictor-746949d687-t525f_882f2be3-ade1-4f59-a2db-a37617e8279d/storage-initializer/0.log"
May 20 15:50:08.762195 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:08.762173    2570 generic.go:358] "Generic (PLEG): container finished" podID="882f2be3-ade1-4f59-a2db-a37617e8279d" containerID="092aaa38c3890651cecac95d947b92e34a410439dfeb13f87d4ef939460a7a1a" exitCode=1
May 20 15:50:08.762277 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:08.762247    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f" event={"ID":"882f2be3-ade1-4f59-a2db-a37617e8279d","Type":"ContainerDied","Data":"092aaa38c3890651cecac95d947b92e34a410439dfeb13f87d4ef939460a7a1a"}
May 20 15:50:08.762312 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:08.762291    2570 scope.go:117] "RemoveContainer" containerID="c7203e5cac2aba7585437e470b077a6996901f6a19b4e3371d46d483e6ee2636"
May 20 15:50:08.762741 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:08.762722    2570 scope.go:117] "RemoveContainer" containerID="c7203e5cac2aba7585437e470b077a6996901f6a19b4e3371d46d483e6ee2636"
May 20 15:50:08.773463 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:50:08.773418    2570 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c3463e-predictor-746949d687-t525f_kserve-ci-e2e-test_882f2be3-ade1-4f59-a2db-a37617e8279d_0 in pod sandbox 09961e7c507a0aca1601558bab0c76da070ec3ad578536f9acd08e1fa82e4139 from index: no such id: 'c7203e5cac2aba7585437e470b077a6996901f6a19b4e3371d46d483e6ee2636'" containerID="c7203e5cac2aba7585437e470b077a6996901f6a19b4e3371d46d483e6ee2636"
May 20 15:50:08.773543 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:50:08.773488    2570 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c3463e-predictor-746949d687-t525f_kserve-ci-e2e-test_882f2be3-ade1-4f59-a2db-a37617e8279d_0 in pod sandbox 09961e7c507a0aca1601558bab0c76da070ec3ad578536f9acd08e1fa82e4139 from index: no such id: 'c7203e5cac2aba7585437e470b077a6996901f6a19b4e3371d46d483e6ee2636'; Skipping pod \"isvc-secondary-c3463e-predictor-746949d687-t525f_kserve-ci-e2e-test(882f2be3-ade1-4f59-a2db-a37617e8279d)\"" logger="UnhandledError"
May 20 15:50:08.774878 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:50:08.774856    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-c3463e-predictor-746949d687-t525f_kserve-ci-e2e-test(882f2be3-ade1-4f59-a2db-a37617e8279d)\"" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f" podUID="882f2be3-ade1-4f59-a2db-a37617e8279d"
May 20 15:50:09.766289 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:09.766257    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c3463e-predictor-746949d687-t525f_882f2be3-ade1-4f59-a2db-a37617e8279d/storage-initializer/1.log"
May 20 15:50:14.313756 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.313702    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"]
May 20 15:50:14.381559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.381526    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"]
May 20 15:50:14.381927 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.381886    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container" containerID="cri-o://933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0" gracePeriod=30
May 20 15:50:14.382023 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.381946    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kube-rbac-proxy" containerID="cri-o://52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163" gracePeriod=30
May 20 15:50:14.436785 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.436746    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"]
May 20 15:50:14.446184 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.446141    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.451298 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.451228    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-eec619\""
May 20 15:50:14.451850 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.451820    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-eec619-dockercfg-f4vmh\""
May 20 15:50:14.452020 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.451998    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-eec619-predictor-serving-cert\""
May 20 15:50:14.452105 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.452005    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-eec619-kube-rbac-proxy-sar-config\""
May 20 15:50:14.454536 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.454506    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"]
May 20 15:50:14.467602 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.467559    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
May 20 15:50:14.504166 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.504141    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c3463e-predictor-746949d687-t525f_882f2be3-ade1-4f59-a2db-a37617e8279d/storage-initializer/1.log"
May 20 15:50:14.504287 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.504214    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:50:14.565970 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.565864    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7gn5\" (UniqueName: \"kubernetes.io/projected/00027eda-57a3-456a-832a-433dd30691fc-kube-api-access-b7gn5\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.565970 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.565912    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00027eda-57a3-456a-832a-433dd30691fc-proxy-tls\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.566202 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.566032    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-eec619-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-isvc-init-fail-eec619-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.566202 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.566095    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-cabundle-cert\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.566278 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.566198    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00027eda-57a3-456a-832a-433dd30691fc-kserve-provision-location\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.667537 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.667500    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-cabundle-cert\") pod \"882f2be3-ade1-4f59-a2db-a37617e8279d\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") "
May 20 15:50:14.667537 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.667548    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-isvc-secondary-c3463e-kube-rbac-proxy-sar-config\") pod \"882f2be3-ade1-4f59-a2db-a37617e8279d\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") "
May 20 15:50:14.667792 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.667587    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/882f2be3-ade1-4f59-a2db-a37617e8279d-proxy-tls\") pod \"882f2be3-ade1-4f59-a2db-a37617e8279d\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") "
May 20 15:50:14.667792 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.667637    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlsn6\" (UniqueName: \"kubernetes.io/projected/882f2be3-ade1-4f59-a2db-a37617e8279d-kube-api-access-dlsn6\") pod \"882f2be3-ade1-4f59-a2db-a37617e8279d\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") "
May 20 15:50:14.667792 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.667781    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/882f2be3-ade1-4f59-a2db-a37617e8279d-kserve-provision-location\") pod \"882f2be3-ade1-4f59-a2db-a37617e8279d\" (UID: \"882f2be3-ade1-4f59-a2db-a37617e8279d\") "
May 20 15:50:14.667956 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.667927    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b7gn5\" (UniqueName: \"kubernetes.io/projected/00027eda-57a3-456a-832a-433dd30691fc-kube-api-access-b7gn5\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.668013 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.667986    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00027eda-57a3-456a-832a-433dd30691fc-proxy-tls\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.668068 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.667973    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "882f2be3-ade1-4f59-a2db-a37617e8279d" (UID: "882f2be3-ade1-4f59-a2db-a37617e8279d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:50:14.668124 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668072    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-eec619-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-isvc-init-fail-eec619-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.668183 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668118    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-isvc-secondary-c3463e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-c3463e-kube-rbac-proxy-sar-config") pod "882f2be3-ade1-4f59-a2db-a37617e8279d" (UID: "882f2be3-ade1-4f59-a2db-a37617e8279d"). InnerVolumeSpecName "isvc-secondary-c3463e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:50:14.668183 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668129    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-cabundle-cert\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.668287 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668267    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00027eda-57a3-456a-832a-433dd30691fc-kserve-provision-location\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.668441 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668424    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-cabundle-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:14.668527 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668447    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/882f2be3-ade1-4f59-a2db-a37617e8279d-isvc-secondary-c3463e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:14.668590 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668555    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882f2be3-ade1-4f59-a2db-a37617e8279d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "882f2be3-ade1-4f59-a2db-a37617e8279d" (UID: "882f2be3-ade1-4f59-a2db-a37617e8279d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:50:14.668802 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668779    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00027eda-57a3-456a-832a-433dd30691fc-kserve-provision-location\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.668984 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668960    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-eec619-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-isvc-init-fail-eec619-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.669025 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.668984    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-cabundle-cert\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.670073 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.670048    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882f2be3-ade1-4f59-a2db-a37617e8279d-kube-api-access-dlsn6" (OuterVolumeSpecName: "kube-api-access-dlsn6") pod "882f2be3-ade1-4f59-a2db-a37617e8279d" (UID: "882f2be3-ade1-4f59-a2db-a37617e8279d"). InnerVolumeSpecName "kube-api-access-dlsn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:50:14.670178 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.670083    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882f2be3-ade1-4f59-a2db-a37617e8279d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "882f2be3-ade1-4f59-a2db-a37617e8279d" (UID: "882f2be3-ade1-4f59-a2db-a37617e8279d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:50:14.670424 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.670407    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00027eda-57a3-456a-832a-433dd30691fc-proxy-tls\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.676136 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.676116    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7gn5\" (UniqueName: \"kubernetes.io/projected/00027eda-57a3-456a-832a-433dd30691fc-kube-api-access-b7gn5\") pod \"isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") " pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.769069 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.769026    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/882f2be3-ade1-4f59-a2db-a37617e8279d-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:14.769069 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.769059    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlsn6\" (UniqueName: \"kubernetes.io/projected/882f2be3-ade1-4f59-a2db-a37617e8279d-kube-api-access-dlsn6\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:14.769069 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.769070    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/882f2be3-ade1-4f59-a2db-a37617e8279d-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:14.769316 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.769171    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:14.786712 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.786685    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c3463e-predictor-746949d687-t525f_882f2be3-ade1-4f59-a2db-a37617e8279d/storage-initializer/1.log"
May 20 15:50:14.786882 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.786791    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f" event={"ID":"882f2be3-ade1-4f59-a2db-a37617e8279d","Type":"ContainerDied","Data":"09961e7c507a0aca1601558bab0c76da070ec3ad578536f9acd08e1fa82e4139"}
May 20 15:50:14.786882 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.786833    2570 scope.go:117] "RemoveContainer" containerID="092aaa38c3890651cecac95d947b92e34a410439dfeb13f87d4ef939460a7a1a"
May 20 15:50:14.786882 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.786834    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"
May 20 15:50:14.788936 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.788913    2570 generic.go:358] "Generic (PLEG): container finished" podID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerID="52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163" exitCode=2
May 20 15:50:14.789058 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.788973    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" event={"ID":"3425f3d9-1734-4ad9-bec4-d94e85c206ec","Type":"ContainerDied","Data":"52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163"}
May 20 15:50:14.829046 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.829003    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"]
May 20 15:50:14.833075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.833034    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c3463e-predictor-746949d687-t525f"]
May 20 15:50:14.913911 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:14.913886    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"]
May 20 15:50:14.916494 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:50:14.916465    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00027eda_57a3_456a_832a_433dd30691fc.slice/crio-1015efad4c3f19555027a9daac14faacb877e6e89e00c388c0f9e695b5277244 WatchSource:0}: Error finding container 1015efad4c3f19555027a9daac14faacb877e6e89e00c388c0f9e695b5277244: Status 404 returned error can't find the container with id 1015efad4c3f19555027a9daac14faacb877e6e89e00c388c0f9e695b5277244
May 20 15:50:15.760550 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:15.760517    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="882f2be3-ade1-4f59-a2db-a37617e8279d" path="/var/lib/kubelet/pods/882f2be3-ade1-4f59-a2db-a37617e8279d/volumes"
May 20 15:50:15.794026 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:15.793990    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq" event={"ID":"00027eda-57a3-456a-832a-433dd30691fc","Type":"ContainerStarted","Data":"431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd"}
May 20 15:50:15.794026 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:15.794027    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq" event={"ID":"00027eda-57a3-456a-832a-433dd30691fc","Type":"ContainerStarted","Data":"1015efad4c3f19555027a9daac14faacb877e6e89e00c388c0f9e695b5277244"}
May 20 15:50:19.028849 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.028822    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:50:19.207666 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.207580    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm982\" (UniqueName: \"kubernetes.io/projected/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kube-api-access-qm982\") pod \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") "
May 20 15:50:19.207666 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.207663    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kserve-provision-location\") pod \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") "
May 20 15:50:19.207857 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.207714    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3425f3d9-1734-4ad9-bec4-d94e85c206ec-isvc-primary-c3463e-kube-rbac-proxy-sar-config\") pod \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") "
May 20 15:50:19.207857 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.207756    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3425f3d9-1734-4ad9-bec4-d94e85c206ec-proxy-tls\") pod \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\" (UID: \"3425f3d9-1734-4ad9-bec4-d94e85c206ec\") "
May 20 15:50:19.208019 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.207995    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3425f3d9-1734-4ad9-bec4-d94e85c206ec" (UID: "3425f3d9-1734-4ad9-bec4-d94e85c206ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:50:19.208097 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.208048    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3425f3d9-1734-4ad9-bec4-d94e85c206ec-isvc-primary-c3463e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-c3463e-kube-rbac-proxy-sar-config") pod "3425f3d9-1734-4ad9-bec4-d94e85c206ec" (UID: "3425f3d9-1734-4ad9-bec4-d94e85c206ec"). InnerVolumeSpecName "isvc-primary-c3463e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:50:19.209752 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.209731    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kube-api-access-qm982" (OuterVolumeSpecName: "kube-api-access-qm982") pod "3425f3d9-1734-4ad9-bec4-d94e85c206ec" (UID: "3425f3d9-1734-4ad9-bec4-d94e85c206ec"). InnerVolumeSpecName "kube-api-access-qm982". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:50:19.209925 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.209906    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3425f3d9-1734-4ad9-bec4-d94e85c206ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3425f3d9-1734-4ad9-bec4-d94e85c206ec" (UID: "3425f3d9-1734-4ad9-bec4-d94e85c206ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:50:19.308767 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.308715    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:19.308767 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.308763    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-c3463e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3425f3d9-1734-4ad9-bec4-d94e85c206ec-isvc-primary-c3463e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:19.308767 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.308779    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3425f3d9-1734-4ad9-bec4-d94e85c206ec-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:19.308984 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.308792    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm982\" (UniqueName: \"kubernetes.io/projected/3425f3d9-1734-4ad9-bec4-d94e85c206ec-kube-api-access-qm982\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:19.811593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.811560    2570 generic.go:358] "Generic (PLEG): container finished" podID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerID="933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0" exitCode=0
May 20 15:50:19.811751 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.811647    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"
May 20 15:50:19.811751 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.811652    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" event={"ID":"3425f3d9-1734-4ad9-bec4-d94e85c206ec","Type":"ContainerDied","Data":"933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0"}
May 20 15:50:19.811751 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.811691    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4" event={"ID":"3425f3d9-1734-4ad9-bec4-d94e85c206ec","Type":"ContainerDied","Data":"0cbfe5e7ce02b269e1c554720e0c02357a820a56e5d395cf8fb334c98d066c0f"}
May 20 15:50:19.811751 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.811707    2570 scope.go:117] "RemoveContainer" containerID="52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163"
May 20 15:50:19.819513 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.819496    2570 scope.go:117] "RemoveContainer" containerID="933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0"
May 20 15:50:19.826574 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.826555    2570 scope.go:117] "RemoveContainer" containerID="f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959"
May 20 15:50:19.831257 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.831235    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"]
May 20 15:50:19.834918 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.834901    2570 scope.go:117] "RemoveContainer" containerID="52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163"
May 20 15:50:19.835069 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.835054    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c3463e-predictor-86cdb4986d-qqmd4"]
May 20 15:50:19.835194 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:50:19.835176    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163\": container with ID starting with 52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163 not found: ID does not exist" containerID="52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163"
May 20 15:50:19.835237 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.835205    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163"} err="failed to get container status \"52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163\": rpc error: code = NotFound desc = could not find container \"52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163\": container with ID starting with 52780d4641d591056242ee58ad0c93863f43e6eab62a66e7c06c214c85658163 not found: ID does not exist"
May 20 15:50:19.835237 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.835227    2570 scope.go:117] "RemoveContainer" containerID="933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0"
May 20 15:50:19.835474 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:50:19.835451    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0\": container with ID starting with 933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0 not found: ID does not exist" containerID="933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0"
May 20 15:50:19.835528 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.835487    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0"} err="failed to get container status \"933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0\": rpc error: code = NotFound desc = could not find container \"933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0\": container with ID starting with 933195e62f66475f62e12915c8d0005a2f3e671599442dc211280ce3279799a0 not found: ID does not exist"
May 20 15:50:19.835528 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.835504    2570 scope.go:117] "RemoveContainer" containerID="f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959"
May 20 15:50:19.835723 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:50:19.835705    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959\": container with ID starting with f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959 not found: ID does not exist" containerID="f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959"
May 20 15:50:19.835774 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:19.835732    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959"} err="failed to get container status \"f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959\": rpc error: code = NotFound desc = could not find container \"f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959\": container with ID starting with f1e404e35bff9fd54c37cb3889a3772c37151b37f43e652707ec133e9ab78959 not found: ID does not exist"
May 20 15:50:21.761700 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:21.761660    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" path="/var/lib/kubelet/pods/3425f3d9-1734-4ad9-bec4-d94e85c206ec/volumes"
May 20 15:50:22.823905 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:22.823868    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq_00027eda-57a3-456a-832a-433dd30691fc/storage-initializer/0.log"
May 20 15:50:22.823905 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:22.823908    2570 generic.go:358] "Generic (PLEG): container finished" podID="00027eda-57a3-456a-832a-433dd30691fc" containerID="431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd" exitCode=1
May 20 15:50:22.824352 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:22.823985    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq" event={"ID":"00027eda-57a3-456a-832a-433dd30691fc","Type":"ContainerDied","Data":"431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd"}
May 20 15:50:23.828305 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:23.828275    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq_00027eda-57a3-456a-832a-433dd30691fc/storage-initializer/0.log"
May 20 15:50:23.828713 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:23.828411    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq" event={"ID":"00027eda-57a3-456a-832a-433dd30691fc","Type":"ContainerStarted","Data":"77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7"}
May 20 15:50:24.426744 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:24.426705    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"]
May 20 15:50:24.832635 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:24.832569    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq" podUID="00027eda-57a3-456a-832a-433dd30691fc" containerName="storage-initializer" containerID="cri-o://77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7" gracePeriod=30
May 20 15:50:25.124025 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.123936    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"]
May 20 15:50:25.124388 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124357    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="storage-initializer"
May 20 15:50:25.124436 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124391    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="storage-initializer"
May 20 15:50:25.124436 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124408    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container"
May 20 15:50:25.124436 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124414    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container"
May 20 15:50:25.124436 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124426    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882f2be3-ade1-4f59-a2db-a37617e8279d" containerName="storage-initializer"
May 20 15:50:25.124436 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124431    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="882f2be3-ade1-4f59-a2db-a37617e8279d" containerName="storage-initializer"
May 20 15:50:25.124593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124440    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882f2be3-ade1-4f59-a2db-a37617e8279d" containerName="storage-initializer"
May 20 15:50:25.124593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124445    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="882f2be3-ade1-4f59-a2db-a37617e8279d" containerName="storage-initializer"
May 20 15:50:25.124593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124453    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kube-rbac-proxy"
May 20 15:50:25.124593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124458    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kube-rbac-proxy"
May 20 15:50:25.124593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124517    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kube-rbac-proxy"
May 20 15:50:25.124593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124526    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="882f2be3-ade1-4f59-a2db-a37617e8279d" containerName="storage-initializer"
May 20 15:50:25.124593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124533    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="3425f3d9-1734-4ad9-bec4-d94e85c206ec" containerName="kserve-container"
May 20 15:50:25.124593 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.124541    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="882f2be3-ade1-4f59-a2db-a37617e8279d" containerName="storage-initializer"
May 20 15:50:25.127874 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.127857    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.130300 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.130261    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qxvlr\""
May 20 15:50:25.130466 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.130405    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-312a7-predictor-serving-cert\""
May 20 15:50:25.130525 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.130408    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-312a7-kube-rbac-proxy-sar-config\""
May 20 15:50:25.138225 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.138199    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"]
May 20 15:50:25.163057 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.163020    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4rj\" (UniqueName: \"kubernetes.io/projected/e10254b7-f583-44f7-a7e5-a2184551333d-kube-api-access-6j4rj\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.163245 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.163066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-312a7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10254b7-f583-44f7-a7e5-a2184551333d-raw-sklearn-312a7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.163245 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.163124    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10254b7-f583-44f7-a7e5-a2184551333d-proxy-tls\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.163245 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.163171    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10254b7-f583-44f7-a7e5-a2184551333d-kserve-provision-location\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.264236 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.264194    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4rj\" (UniqueName: \"kubernetes.io/projected/e10254b7-f583-44f7-a7e5-a2184551333d-kube-api-access-6j4rj\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.264452 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.264242    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-312a7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10254b7-f583-44f7-a7e5-a2184551333d-raw-sklearn-312a7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.264452 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.264290    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10254b7-f583-44f7-a7e5-a2184551333d-proxy-tls\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.264452 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.264340    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10254b7-f583-44f7-a7e5-a2184551333d-kserve-provision-location\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.264748 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.264726    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10254b7-f583-44f7-a7e5-a2184551333d-kserve-provision-location\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.265011 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.264988    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-312a7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10254b7-f583-44f7-a7e5-a2184551333d-raw-sklearn-312a7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.266880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.266858    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10254b7-f583-44f7-a7e5-a2184551333d-proxy-tls\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.272329 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.272300    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4rj\" (UniqueName: \"kubernetes.io/projected/e10254b7-f583-44f7-a7e5-a2184551333d-kube-api-access-6j4rj\") pod \"raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") " pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.439621 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.439523    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:25.579046 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.579019    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"]
May 20 15:50:25.581567 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:50:25.581534    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10254b7_f583_44f7_a7e5_a2184551333d.slice/crio-5da589420907fd2061237b5d892397a95fde861cda8e76efe0d42b75c955bf9b WatchSource:0}: Error finding container 5da589420907fd2061237b5d892397a95fde861cda8e76efe0d42b75c955bf9b: Status 404 returned error can't find the container with id 5da589420907fd2061237b5d892397a95fde861cda8e76efe0d42b75c955bf9b
May 20 15:50:25.838017 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.837977    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" event={"ID":"e10254b7-f583-44f7-a7e5-a2184551333d","Type":"ContainerStarted","Data":"b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc"}
May 20 15:50:25.838017 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:25.838022    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" event={"ID":"e10254b7-f583-44f7-a7e5-a2184551333d","Type":"ContainerStarted","Data":"5da589420907fd2061237b5d892397a95fde861cda8e76efe0d42b75c955bf9b"}
May 20 15:50:26.583421 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.583401    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq_00027eda-57a3-456a-832a-433dd30691fc/storage-initializer/1.log"
May 20 15:50:26.583781 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.583766    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq_00027eda-57a3-456a-832a-433dd30691fc/storage-initializer/0.log"
May 20 15:50:26.583840 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.583830    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:26.674770 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.674686    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00027eda-57a3-456a-832a-433dd30691fc-kserve-provision-location\") pod \"00027eda-57a3-456a-832a-433dd30691fc\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") "
May 20 15:50:26.674770 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.674748    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00027eda-57a3-456a-832a-433dd30691fc-proxy-tls\") pod \"00027eda-57a3-456a-832a-433dd30691fc\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") "
May 20 15:50:26.674978 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.674803    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7gn5\" (UniqueName: \"kubernetes.io/projected/00027eda-57a3-456a-832a-433dd30691fc-kube-api-access-b7gn5\") pod \"00027eda-57a3-456a-832a-433dd30691fc\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") "
May 20 15:50:26.674978 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.674824    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-cabundle-cert\") pod \"00027eda-57a3-456a-832a-433dd30691fc\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") "
May 20 15:50:26.674978 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.674897    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-eec619-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-isvc-init-fail-eec619-kube-rbac-proxy-sar-config\") pod \"00027eda-57a3-456a-832a-433dd30691fc\" (UID: \"00027eda-57a3-456a-832a-433dd30691fc\") "
May 20 15:50:26.674978 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.674936    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00027eda-57a3-456a-832a-433dd30691fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00027eda-57a3-456a-832a-433dd30691fc" (UID: "00027eda-57a3-456a-832a-433dd30691fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:50:26.675241 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.675153    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00027eda-57a3-456a-832a-433dd30691fc-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:26.675241 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.675212    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00027eda-57a3-456a-832a-433dd30691fc" (UID: "00027eda-57a3-456a-832a-433dd30691fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:50:26.675337 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.675256    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-isvc-init-fail-eec619-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-eec619-kube-rbac-proxy-sar-config") pod "00027eda-57a3-456a-832a-433dd30691fc" (UID: "00027eda-57a3-456a-832a-433dd30691fc"). InnerVolumeSpecName "isvc-init-fail-eec619-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:50:26.676858 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.676830    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00027eda-57a3-456a-832a-433dd30691fc-kube-api-access-b7gn5" (OuterVolumeSpecName: "kube-api-access-b7gn5") pod "00027eda-57a3-456a-832a-433dd30691fc" (UID: "00027eda-57a3-456a-832a-433dd30691fc"). InnerVolumeSpecName "kube-api-access-b7gn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:50:26.676953 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.676880    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00027eda-57a3-456a-832a-433dd30691fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00027eda-57a3-456a-832a-433dd30691fc" (UID: "00027eda-57a3-456a-832a-433dd30691fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:50:26.775647 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.775613    2570 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-eec619-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-isvc-init-fail-eec619-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:26.775647 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.775642    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00027eda-57a3-456a-832a-433dd30691fc-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:26.775647 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.775653    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7gn5\" (UniqueName: \"kubernetes.io/projected/00027eda-57a3-456a-832a-433dd30691fc-kube-api-access-b7gn5\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:26.775852 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.775663    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00027eda-57a3-456a-832a-433dd30691fc-cabundle-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:50:26.842374 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.842350    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq_00027eda-57a3-456a-832a-433dd30691fc/storage-initializer/1.log"
May 20 15:50:26.842762 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.842701    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq_00027eda-57a3-456a-832a-433dd30691fc/storage-initializer/0.log"
May 20 15:50:26.842762 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.842732    2570 generic.go:358] "Generic (PLEG): container finished" podID="00027eda-57a3-456a-832a-433dd30691fc" containerID="77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7" exitCode=1
May 20 15:50:26.842833 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.842823    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"
May 20 15:50:26.842912 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.842816    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq" event={"ID":"00027eda-57a3-456a-832a-433dd30691fc","Type":"ContainerDied","Data":"77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7"}
May 20 15:50:26.842948 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.842931    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq" event={"ID":"00027eda-57a3-456a-832a-433dd30691fc","Type":"ContainerDied","Data":"1015efad4c3f19555027a9daac14faacb877e6e89e00c388c0f9e695b5277244"}
May 20 15:50:26.842978 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.842946    2570 scope.go:117] "RemoveContainer" containerID="77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7"
May 20 15:50:26.851525 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.851507    2570 scope.go:117] "RemoveContainer" containerID="431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd"
May 20 15:50:26.858974 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.858955    2570 scope.go:117] "RemoveContainer" containerID="77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7"
May 20 15:50:26.859234 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:50:26.859215    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7\": container with ID starting with 77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7 not found: ID does not exist" containerID="77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7"
May 20 15:50:26.859280 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.859243    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7"} err="failed to get container status \"77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7\": rpc error: code = NotFound desc = could not find container \"77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7\": container with ID starting with 77cda936b3c15be0fd85045b2eaa449bd31daefbfbfe2d27fcd84ac8d703d6b7 not found: ID does not exist"
May 20 15:50:26.859280 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.859261    2570 scope.go:117] "RemoveContainer" containerID="431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd"
May 20 15:50:26.859518 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:50:26.859501    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd\": container with ID starting with 431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd not found: ID does not exist" containerID="431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd"
May 20 15:50:26.859568 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.859523    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd"} err="failed to get container status \"431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd\": rpc error: code = NotFound desc = could not find container \"431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd\": container with ID starting with 431219d55142afaf8e1c933299d46757f00afe6f3534a36c9c3bd70517904ebd not found: ID does not exist"
May 20 15:50:26.882681 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.882656    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"]
May 20 15:50:26.886334 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:26.886308    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-eec619-predictor-6bd958d4dc-zbwhq"]
May 20 15:50:27.759796 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:27.759765    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00027eda-57a3-456a-832a-433dd30691fc" path="/var/lib/kubelet/pods/00027eda-57a3-456a-832a-433dd30691fc/volumes"
May 20 15:50:29.854417 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:29.854382    2570 generic.go:358] "Generic (PLEG): container finished" podID="e10254b7-f583-44f7-a7e5-a2184551333d" containerID="b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc" exitCode=0
May 20 15:50:29.854779 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:29.854436    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" event={"ID":"e10254b7-f583-44f7-a7e5-a2184551333d","Type":"ContainerDied","Data":"b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc"}
May 20 15:50:30.859789 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:30.859753    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" event={"ID":"e10254b7-f583-44f7-a7e5-a2184551333d","Type":"ContainerStarted","Data":"efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9"}
May 20 15:50:30.859789 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:30.859798    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" event={"ID":"e10254b7-f583-44f7-a7e5-a2184551333d","Type":"ContainerStarted","Data":"66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045"}
May 20 15:50:30.860281 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:30.860081    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:30.860281 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:30.860218    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:30.861630 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:30.861602    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 15:50:30.883641 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:30.883588    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podStartSLOduration=5.883571722 podStartE2EDuration="5.883571722s" podCreationTimestamp="2026-05-20 15:50:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:50:30.881611796 +0000 UTC m=+1119.711707718" watchObservedRunningTime="2026-05-20 15:50:30.883571722 +0000 UTC m=+1119.713667643"
May 20 15:50:31.864019 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:31.863973    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 15:50:36.868807 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:36.868780    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:50:36.869350 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:36.869323    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 15:50:46.869971 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:46.869922    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 15:50:56.869581 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:50:56.869539    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 15:51:06.870283 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:06.870183    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 15:51:16.869655 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:16.869609    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 15:51:26.869293 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:26.869254    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 15:51:36.870201 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:36.870168    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:51:44.682036 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.682001    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"]
May 20 15:51:44.682553 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.682357    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" containerID="cri-o://66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045" gracePeriod=30
May 20 15:51:44.682553 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.682424    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kube-rbac-proxy" containerID="cri-o://efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9" gracePeriod=30
May 20 15:51:44.785261 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.785227    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"]
May 20 15:51:44.785639 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.785623    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00027eda-57a3-456a-832a-433dd30691fc" containerName="storage-initializer"
May 20 15:51:44.785689 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.785641    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00027eda-57a3-456a-832a-433dd30691fc" containerName="storage-initializer"
May 20 15:51:44.785731 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.785718    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="00027eda-57a3-456a-832a-433dd30691fc" containerName="storage-initializer"
May 20 15:51:44.785731 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.785727    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="00027eda-57a3-456a-832a-433dd30691fc" containerName="storage-initializer"
May 20 15:51:44.785820 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.785798    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00027eda-57a3-456a-832a-433dd30691fc" containerName="storage-initializer"
May 20 15:51:44.785820 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.785810    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00027eda-57a3-456a-832a-433dd30691fc" containerName="storage-initializer"
May 20 15:51:44.789067 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.789047    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.791432 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.791408    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-77323-predictor-serving-cert\""
May 20 15:51:44.791548 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.791421    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\""
May 20 15:51:44.797623 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.797603    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"]
May 20 15:51:44.873630 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.873588    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96z87\" (UniqueName: \"kubernetes.io/projected/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kube-api-access-96z87\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.873815 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.873755    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b83c05-8f66-405b-a6f3-fae7d47e42b8-proxy-tls\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.873815 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.873793    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kserve-provision-location\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.873968 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.873937    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b83c05-8f66-405b-a6f3-fae7d47e42b8-raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.974480 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.974387    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b83c05-8f66-405b-a6f3-fae7d47e42b8-proxy-tls\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.974480 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.974441    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kserve-provision-location\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.974678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.974512    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b83c05-8f66-405b-a6f3-fae7d47e42b8-raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.974678 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.974564    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-96z87\" (UniqueName: \"kubernetes.io/projected/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kube-api-access-96z87\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.974837 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.974816    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kserve-provision-location\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.975196 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.975176    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b83c05-8f66-405b-a6f3-fae7d47e42b8-raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.976842 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.976822    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b83c05-8f66-405b-a6f3-fae7d47e42b8-proxy-tls\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:44.984434 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:44.984414    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96z87\" (UniqueName: \"kubernetes.io/projected/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kube-api-access-96z87\") pod \"raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:45.098311 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:45.098276    2570 generic.go:358] "Generic (PLEG): container finished" podID="e10254b7-f583-44f7-a7e5-a2184551333d" containerID="efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9" exitCode=2
May 20 15:51:45.098494 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:45.098315    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" event={"ID":"e10254b7-f583-44f7-a7e5-a2184551333d","Type":"ContainerDied","Data":"efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9"}
May 20 15:51:45.101522 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:45.101497    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:45.230075 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:45.230017    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"]
May 20 15:51:45.232742 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:51:45.232712    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b83c05_8f66_405b_a6f3_fae7d47e42b8.slice/crio-105ff53d86d8dcf4c0f0c88dce48d7a2390bf16ddb2db32fe54b8742003aea00 WatchSource:0}: Error finding container 105ff53d86d8dcf4c0f0c88dce48d7a2390bf16ddb2db32fe54b8742003aea00: Status 404 returned error can't find the container with id 105ff53d86d8dcf4c0f0c88dce48d7a2390bf16ddb2db32fe54b8742003aea00
May 20 15:51:46.103506 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:46.103467    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" event={"ID":"03b83c05-8f66-405b-a6f3-fae7d47e42b8","Type":"ContainerStarted","Data":"6d93fd692104f76479d4c4a153a0309579d316bae850b65c4c7ac2661f426d60"}
May 20 15:51:46.103506 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:46.103505    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" event={"ID":"03b83c05-8f66-405b-a6f3-fae7d47e42b8","Type":"ContainerStarted","Data":"105ff53d86d8dcf4c0f0c88dce48d7a2390bf16ddb2db32fe54b8742003aea00"}
May 20 15:51:46.864809 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:46.864765    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
May 20 15:51:46.869846 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:46.869816    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 15:51:49.028071 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.028049    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:51:49.109360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.109262    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10254b7-f583-44f7-a7e5-a2184551333d-kserve-provision-location\") pod \"e10254b7-f583-44f7-a7e5-a2184551333d\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") "
May 20 15:51:49.109360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.109304    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10254b7-f583-44f7-a7e5-a2184551333d-proxy-tls\") pod \"e10254b7-f583-44f7-a7e5-a2184551333d\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") "
May 20 15:51:49.109626 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.109411    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j4rj\" (UniqueName: \"kubernetes.io/projected/e10254b7-f583-44f7-a7e5-a2184551333d-kube-api-access-6j4rj\") pod \"e10254b7-f583-44f7-a7e5-a2184551333d\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") "
May 20 15:51:49.109626 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.109438    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-312a7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10254b7-f583-44f7-a7e5-a2184551333d-raw-sklearn-312a7-kube-rbac-proxy-sar-config\") pod \"e10254b7-f583-44f7-a7e5-a2184551333d\" (UID: \"e10254b7-f583-44f7-a7e5-a2184551333d\") "
May 20 15:51:49.109753 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.109683    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10254b7-f583-44f7-a7e5-a2184551333d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e10254b7-f583-44f7-a7e5-a2184551333d" (UID: "e10254b7-f583-44f7-a7e5-a2184551333d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:51:49.109874 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.109837    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10254b7-f583-44f7-a7e5-a2184551333d-raw-sklearn-312a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-312a7-kube-rbac-proxy-sar-config") pod "e10254b7-f583-44f7-a7e5-a2184551333d" (UID: "e10254b7-f583-44f7-a7e5-a2184551333d"). InnerVolumeSpecName "raw-sklearn-312a7-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:51:49.111835 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.111798    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10254b7-f583-44f7-a7e5-a2184551333d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e10254b7-f583-44f7-a7e5-a2184551333d" (UID: "e10254b7-f583-44f7-a7e5-a2184551333d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:51:49.111969 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.111873    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10254b7-f583-44f7-a7e5-a2184551333d-kube-api-access-6j4rj" (OuterVolumeSpecName: "kube-api-access-6j4rj") pod "e10254b7-f583-44f7-a7e5-a2184551333d" (UID: "e10254b7-f583-44f7-a7e5-a2184551333d"). InnerVolumeSpecName "kube-api-access-6j4rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:51:49.114800 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.114772    2570 generic.go:358] "Generic (PLEG): container finished" podID="e10254b7-f583-44f7-a7e5-a2184551333d" containerID="66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045" exitCode=0
May 20 15:51:49.114926 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.114847    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" event={"ID":"e10254b7-f583-44f7-a7e5-a2184551333d","Type":"ContainerDied","Data":"66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045"}
May 20 15:51:49.114926 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.114892    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4" event={"ID":"e10254b7-f583-44f7-a7e5-a2184551333d","Type":"ContainerDied","Data":"5da589420907fd2061237b5d892397a95fde861cda8e76efe0d42b75c955bf9b"}
May 20 15:51:49.114926 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.114858    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"
May 20 15:51:49.114926 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.114911    2570 scope.go:117] "RemoveContainer" containerID="efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9"
May 20 15:51:49.116443 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.116416    2570 generic.go:358] "Generic (PLEG): container finished" podID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerID="6d93fd692104f76479d4c4a153a0309579d316bae850b65c4c7ac2661f426d60" exitCode=0
May 20 15:51:49.116557 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.116475    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" event={"ID":"03b83c05-8f66-405b-a6f3-fae7d47e42b8","Type":"ContainerDied","Data":"6d93fd692104f76479d4c4a153a0309579d316bae850b65c4c7ac2661f426d60"}
May 20 15:51:49.125316 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.125295    2570 scope.go:117] "RemoveContainer" containerID="66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045"
May 20 15:51:49.135038 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.135016    2570 scope.go:117] "RemoveContainer" containerID="b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc"
May 20 15:51:49.144644 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.144626    2570 scope.go:117] "RemoveContainer" containerID="efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9"
May 20 15:51:49.144942 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:51:49.144918    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9\": container with ID starting with efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9 not found: ID does not exist" containerID="efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9"
May 20 15:51:49.144982 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.144955    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9"} err="failed to get container status \"efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9\": rpc error: code = NotFound desc = could not find container \"efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9\": container with ID starting with efcac0607cf70368fca1d4807d68c31f980149527b1b3386c28dd369f35c0eb9 not found: ID does not exist"
May 20 15:51:49.144982 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.144979    2570 scope.go:117] "RemoveContainer" containerID="66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045"
May 20 15:51:49.145251 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:51:49.145231    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045\": container with ID starting with 66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045 not found: ID does not exist" containerID="66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045"
May 20 15:51:49.145315 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.145256    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045"} err="failed to get container status \"66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045\": rpc error: code = NotFound desc = could not find container \"66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045\": container with ID starting with 66def13479698f4079a52c7893376a00ae21a64daca9e8edd667a56bed392045 not found: ID does not exist"
May 20 15:51:49.145315 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.145273    2570 scope.go:117] "RemoveContainer" containerID="b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc"
May 20 15:51:49.145527 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:51:49.145507    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc\": container with ID starting with b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc not found: ID does not exist" containerID="b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc"
May 20 15:51:49.145571 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.145531    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc"} err="failed to get container status \"b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc\": rpc error: code = NotFound desc = could not find container \"b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc\": container with ID starting with b4006c8c059bbdcf425ebc4aa35d7ec8ad10267be5add64c805e670626769edc not found: ID does not exist"
May 20 15:51:49.153417 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.153390    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"]
May 20 15:51:49.157296 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.157265    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-312a7-predictor-666cb5fbf5-g4qg4"]
May 20 15:51:49.210569 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.210538    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j4rj\" (UniqueName: \"kubernetes.io/projected/e10254b7-f583-44f7-a7e5-a2184551333d-kube-api-access-6j4rj\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:51:49.210569 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.210566    2570 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-312a7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10254b7-f583-44f7-a7e5-a2184551333d-raw-sklearn-312a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:51:49.210767 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.210579    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10254b7-f583-44f7-a7e5-a2184551333d-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:51:49.210767 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.210591    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10254b7-f583-44f7-a7e5-a2184551333d-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:51:49.759477 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:49.759440    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" path="/var/lib/kubelet/pods/e10254b7-f583-44f7-a7e5-a2184551333d/volumes"
May 20 15:51:50.122344 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:50.122301    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" event={"ID":"03b83c05-8f66-405b-a6f3-fae7d47e42b8","Type":"ContainerStarted","Data":"0cabc2051c576f5a9da0f57fef12ff37c5ffa7ee00ce71419d6a5b676e7a6744"}
May 20 15:51:50.122344 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:50.122352    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" event={"ID":"03b83c05-8f66-405b-a6f3-fae7d47e42b8","Type":"ContainerStarted","Data":"1cd2ac5fb4c9dac06ee440c588a7a04828840f570bfbfbc950922c1b1ac3a707"}
May 20 15:51:50.122869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:50.122685    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:50.122869 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:50.122847    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:50.124131 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:50.124102    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 15:51:50.143731 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:50.143680    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podStartSLOduration=6.143663673 podStartE2EDuration="6.143663673s" podCreationTimestamp="2026-05-20 15:51:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:51:50.141286415 +0000 UTC m=+1198.971382360" watchObservedRunningTime="2026-05-20 15:51:50.143663673 +0000 UTC m=+1198.973759595"
May 20 15:51:51.127183 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:51.127147    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 15:51:51.717150 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:51.717122    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:51:51.719733 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:51.719712    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:51:56.131320 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:56.131290    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:51:56.131920 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:51:56.131888    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 15:52:06.132278 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:52:06.132216    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 15:52:16.132079 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:52:16.132037    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 15:52:26.132231 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:52:26.132190    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 15:52:36.132396 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:52:36.132292    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 15:52:46.132360 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:52:46.132314    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 15:52:56.133260 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:52:56.133228    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:53:06.152812 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:06.152773    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"]
May 20 15:53:06.153309 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:06.153233    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container" containerID="cri-o://1cd2ac5fb4c9dac06ee440c588a7a04828840f570bfbfbc950922c1b1ac3a707" gracePeriod=30
May 20 15:53:06.153395 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:06.153309    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kube-rbac-proxy" containerID="cri-o://0cabc2051c576f5a9da0f57fef12ff37c5ffa7ee00ce71419d6a5b676e7a6744" gracePeriod=30
May 20 15:53:06.368996 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:06.368959    2570 generic.go:358] "Generic (PLEG): container finished" podID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerID="0cabc2051c576f5a9da0f57fef12ff37c5ffa7ee00ce71419d6a5b676e7a6744" exitCode=2
May 20 15:53:06.369172 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:06.369037    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" event={"ID":"03b83c05-8f66-405b-a6f3-fae7d47e42b8","Type":"ContainerDied","Data":"0cabc2051c576f5a9da0f57fef12ff37c5ffa7ee00ce71419d6a5b676e7a6744"}
May 20 15:53:08.512700 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.512665    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8kbs2/must-gather-zgf5k"]
May 20 15:53:08.513132 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.513046    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="storage-initializer"
May 20 15:53:08.513132 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.513057    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="storage-initializer"
May 20 15:53:08.513132 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.513068    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kube-rbac-proxy"
May 20 15:53:08.513132 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.513073    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kube-rbac-proxy"
May 20 15:53:08.513132 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.513085    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container"
May 20 15:53:08.513132 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.513092    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container"
May 20 15:53:08.513354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.513165    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kserve-container"
May 20 15:53:08.513354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.513176    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="e10254b7-f583-44f7-a7e5-a2184551333d" containerName="kube-rbac-proxy"
May 20 15:53:08.516415 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.516394    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:08.519059 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.519033    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8kbs2\"/\"openshift-service-ca.crt\""
May 20 15:53:08.519772 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.519752    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8kbs2\"/\"default-dockercfg-kx68d\""
May 20 15:53:08.519838 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.519771    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8kbs2\"/\"kube-root-ca.crt\""
May 20 15:53:08.525312 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.525274    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kbs2/must-gather-zgf5k"]
May 20 15:53:08.623629 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.623585    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6xqr\" (UniqueName: \"kubernetes.io/projected/44b94679-1ef8-454f-a3bc-61bcc8ecb777-kube-api-access-n6xqr\") pod \"must-gather-zgf5k\" (UID: \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\") " pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:08.623810 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.623734    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44b94679-1ef8-454f-a3bc-61bcc8ecb777-must-gather-output\") pod \"must-gather-zgf5k\" (UID: \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\") " pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:08.724928 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.724883    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44b94679-1ef8-454f-a3bc-61bcc8ecb777-must-gather-output\") pod \"must-gather-zgf5k\" (UID: \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\") " pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:08.725098 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.725055    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n6xqr\" (UniqueName: \"kubernetes.io/projected/44b94679-1ef8-454f-a3bc-61bcc8ecb777-kube-api-access-n6xqr\") pod \"must-gather-zgf5k\" (UID: \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\") " pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:08.725289 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.725267    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44b94679-1ef8-454f-a3bc-61bcc8ecb777-must-gather-output\") pod \"must-gather-zgf5k\" (UID: \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\") " pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:08.734090 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.734061    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6xqr\" (UniqueName: \"kubernetes.io/projected/44b94679-1ef8-454f-a3bc-61bcc8ecb777-kube-api-access-n6xqr\") pod \"must-gather-zgf5k\" (UID: \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\") " pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:08.833025 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.832985    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:08.972657 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:08.972623    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kbs2/must-gather-zgf5k"]
May 20 15:53:08.975531 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:53:08.975494    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b94679_1ef8_454f_a3bc_61bcc8ecb777.slice/crio-9acfad585bc10a82d74d0ed0d66c69c02071fd58a3eb7c8b0bceb4970cc96cff WatchSource:0}: Error finding container 9acfad585bc10a82d74d0ed0d66c69c02071fd58a3eb7c8b0bceb4970cc96cff: Status 404 returned error can't find the container with id 9acfad585bc10a82d74d0ed0d66c69c02071fd58a3eb7c8b0bceb4970cc96cff
May 20 15:53:09.379785 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:09.379743    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" event={"ID":"44b94679-1ef8-454f-a3bc-61bcc8ecb777","Type":"ContainerStarted","Data":"9acfad585bc10a82d74d0ed0d66c69c02071fd58a3eb7c8b0bceb4970cc96cff"}
May 20 15:53:11.127848 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:11.127802    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 20 15:53:13.396581 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:13.396534    2570 generic.go:358] "Generic (PLEG): container finished" podID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerID="1cd2ac5fb4c9dac06ee440c588a7a04828840f570bfbfbc950922c1b1ac3a707" exitCode=0
May 20 15:53:13.397107 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:13.396588    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" event={"ID":"03b83c05-8f66-405b-a6f3-fae7d47e42b8","Type":"ContainerDied","Data":"1cd2ac5fb4c9dac06ee440c588a7a04828840f570bfbfbc950922c1b1ac3a707"}
May 20 15:53:14.865188 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.865158    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:53:14.989866 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.989815    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kserve-provision-location\") pod \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") "
May 20 15:53:14.989866 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.989871    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96z87\" (UniqueName: \"kubernetes.io/projected/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kube-api-access-96z87\") pod \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") "
May 20 15:53:14.990117 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.989955    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b83c05-8f66-405b-a6f3-fae7d47e42b8-proxy-tls\") pod \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") "
May 20 15:53:14.990117 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.990017    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b83c05-8f66-405b-a6f3-fae7d47e42b8-raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\") pod \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\" (UID: \"03b83c05-8f66-405b-a6f3-fae7d47e42b8\") "
May 20 15:53:14.990237 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.990184    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03b83c05-8f66-405b-a6f3-fae7d47e42b8" (UID: "03b83c05-8f66-405b-a6f3-fae7d47e42b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:53:14.990423 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.990401    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kserve-provision-location\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:53:14.990528 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.990499    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b83c05-8f66-405b-a6f3-fae7d47e42b8-raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config") pod "03b83c05-8f66-405b-a6f3-fae7d47e42b8" (UID: "03b83c05-8f66-405b-a6f3-fae7d47e42b8"). InnerVolumeSpecName "raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:53:14.992417 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.992389    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kube-api-access-96z87" (OuterVolumeSpecName: "kube-api-access-96z87") pod "03b83c05-8f66-405b-a6f3-fae7d47e42b8" (UID: "03b83c05-8f66-405b-a6f3-fae7d47e42b8"). InnerVolumeSpecName "kube-api-access-96z87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:53:14.992738 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:14.992709    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b83c05-8f66-405b-a6f3-fae7d47e42b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03b83c05-8f66-405b-a6f3-fae7d47e42b8" (UID: "03b83c05-8f66-405b-a6f3-fae7d47e42b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:53:15.091076 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.090983    2570 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b83c05-8f66-405b-a6f3-fae7d47e42b8-raw-sklearn-runtime-77323-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:53:15.091076 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.091040    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96z87\" (UniqueName: \"kubernetes.io/projected/03b83c05-8f66-405b-a6f3-fae7d47e42b8-kube-api-access-96z87\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:53:15.091076 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.091059    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b83c05-8f66-405b-a6f3-fae7d47e42b8-proxy-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:53:15.406205 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.406119    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"
May 20 15:53:15.406403 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.406126    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs" event={"ID":"03b83c05-8f66-405b-a6f3-fae7d47e42b8","Type":"ContainerDied","Data":"105ff53d86d8dcf4c0f0c88dce48d7a2390bf16ddb2db32fe54b8742003aea00"}
May 20 15:53:15.406403 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.406253    2570 scope.go:117] "RemoveContainer" containerID="0cabc2051c576f5a9da0f57fef12ff37c5ffa7ee00ce71419d6a5b676e7a6744"
May 20 15:53:15.408145 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.408109    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" event={"ID":"44b94679-1ef8-454f-a3bc-61bcc8ecb777","Type":"ContainerStarted","Data":"e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4"}
May 20 15:53:15.408145 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.408139    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" event={"ID":"44b94679-1ef8-454f-a3bc-61bcc8ecb777","Type":"ContainerStarted","Data":"18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21"}
May 20 15:53:15.416448 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.416425    2570 scope.go:117] "RemoveContainer" containerID="1cd2ac5fb4c9dac06ee440c588a7a04828840f570bfbfbc950922c1b1ac3a707"
May 20 15:53:15.425100 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.425076    2570 scope.go:117] "RemoveContainer" containerID="6d93fd692104f76479d4c4a153a0309579d316bae850b65c4c7ac2661f426d60"
May 20 15:53:15.429028 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.428974    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" podStartSLOduration=1.449206682 podStartE2EDuration="7.428957578s" podCreationTimestamp="2026-05-20 15:53:08 +0000 UTC" firstStartedPulling="2026-05-20 15:53:08.977317311 +0000 UTC m=+1277.807413212" lastFinishedPulling="2026-05-20 15:53:14.957068196 +0000 UTC m=+1283.787164108" observedRunningTime="2026-05-20 15:53:15.426248048 +0000 UTC m=+1284.256343968" watchObservedRunningTime="2026-05-20 15:53:15.428957578 +0000 UTC m=+1284.259053499"
May 20 15:53:15.446234 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.446190    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"]
May 20 15:53:15.451472 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.451428    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-77323-predictor-6d8b8f8c49-g92hs"]
May 20 15:53:15.761441 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:15.761342    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" path="/var/lib/kubelet/pods/03b83c05-8f66-405b-a6f3-fae7d47e42b8/volumes"
May 20 15:53:36.486872 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:36.486825    2570 generic.go:358] "Generic (PLEG): container finished" podID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" containerID="18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21" exitCode=0
May 20 15:53:36.487354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:36.486913    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" event={"ID":"44b94679-1ef8-454f-a3bc-61bcc8ecb777","Type":"ContainerDied","Data":"18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21"}
May 20 15:53:36.487354 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:36.487347    2570 scope.go:117] "RemoveContainer" containerID="18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21"
May 20 15:53:36.880592 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:36.880549    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8kbs2_must-gather-zgf5k_44b94679-1ef8-454f-a3bc-61bcc8ecb777/gather/0.log"
May 20 15:53:40.615979 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.615930    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2htxn/must-gather-9dcdm"]
May 20 15:53:40.616530 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.616508    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container"
May 20 15:53:40.616620 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.616535    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container"
May 20 15:53:40.616620 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.616578    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kube-rbac-proxy"
May 20 15:53:40.616620 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.616587    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kube-rbac-proxy"
May 20 15:53:40.616620 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.616603    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="storage-initializer"
May 20 15:53:40.616620 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.616612    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="storage-initializer"
May 20 15:53:40.616884 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.616713    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kube-rbac-proxy"
May 20 15:53:40.616884 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.616730    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="03b83c05-8f66-405b-a6f3-fae7d47e42b8" containerName="kserve-container"
May 20 15:53:40.618775 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.618752    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2htxn/must-gather-9dcdm"
May 20 15:53:40.621134 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.621107    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2htxn\"/\"kube-root-ca.crt\""
May 20 15:53:40.621905 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.621885    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2htxn\"/\"default-dockercfg-z2b6t\""
May 20 15:53:40.622043 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.621958    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2htxn\"/\"openshift-service-ca.crt\""
May 20 15:53:40.626762 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.626736    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2htxn/must-gather-9dcdm"]
May 20 15:53:40.731421 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.731336    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjlj\" (UniqueName: \"kubernetes.io/projected/2544e232-41b9-4494-bdd8-74a0a0d88a59-kube-api-access-4mjlj\") pod \"must-gather-9dcdm\" (UID: \"2544e232-41b9-4494-bdd8-74a0a0d88a59\") " pod="openshift-must-gather-2htxn/must-gather-9dcdm"
May 20 15:53:40.731421 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.731427    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2544e232-41b9-4494-bdd8-74a0a0d88a59-must-gather-output\") pod \"must-gather-9dcdm\" (UID: \"2544e232-41b9-4494-bdd8-74a0a0d88a59\") " pod="openshift-must-gather-2htxn/must-gather-9dcdm"
May 20 15:53:40.832850 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.832807    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjlj\" (UniqueName: \"kubernetes.io/projected/2544e232-41b9-4494-bdd8-74a0a0d88a59-kube-api-access-4mjlj\") pod \"must-gather-9dcdm\" (UID: \"2544e232-41b9-4494-bdd8-74a0a0d88a59\") " pod="openshift-must-gather-2htxn/must-gather-9dcdm"
May 20 15:53:40.832850 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.832849    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2544e232-41b9-4494-bdd8-74a0a0d88a59-must-gather-output\") pod \"must-gather-9dcdm\" (UID: \"2544e232-41b9-4494-bdd8-74a0a0d88a59\") " pod="openshift-must-gather-2htxn/must-gather-9dcdm"
May 20 15:53:40.833190 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.833174    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2544e232-41b9-4494-bdd8-74a0a0d88a59-must-gather-output\") pod \"must-gather-9dcdm\" (UID: \"2544e232-41b9-4494-bdd8-74a0a0d88a59\") " pod="openshift-must-gather-2htxn/must-gather-9dcdm"
May 20 15:53:40.841421 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.841361    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjlj\" (UniqueName: \"kubernetes.io/projected/2544e232-41b9-4494-bdd8-74a0a0d88a59-kube-api-access-4mjlj\") pod \"must-gather-9dcdm\" (UID: \"2544e232-41b9-4494-bdd8-74a0a0d88a59\") " pod="openshift-must-gather-2htxn/must-gather-9dcdm"
May 20 15:53:40.929142 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:40.929039    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2htxn/must-gather-9dcdm"
May 20 15:53:41.062464 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:41.062429    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2htxn/must-gather-9dcdm"]
May 20 15:53:41.065419 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:53:41.065387    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2544e232_41b9_4494_bdd8_74a0a0d88a59.slice/crio-48e9549cec71c6a8dd8056c30a020d3080206b29a80712f178d1692dbfe3acff WatchSource:0}: Error finding container 48e9549cec71c6a8dd8056c30a020d3080206b29a80712f178d1692dbfe3acff: Status 404 returned error can't find the container with id 48e9549cec71c6a8dd8056c30a020d3080206b29a80712f178d1692dbfe3acff
May 20 15:53:41.504007 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:41.503969    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2htxn/must-gather-9dcdm" event={"ID":"2544e232-41b9-4494-bdd8-74a0a0d88a59","Type":"ContainerStarted","Data":"48e9549cec71c6a8dd8056c30a020d3080206b29a80712f178d1692dbfe3acff"}
May 20 15:53:42.509964 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:42.509929    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2htxn/must-gather-9dcdm" event={"ID":"2544e232-41b9-4494-bdd8-74a0a0d88a59","Type":"ContainerStarted","Data":"1be60d1baac1e2112a094841061ed2f0c7b86f8a6c2efc9ec39e306f452caa55"}
May 20 15:53:42.509964 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:42.509969    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2htxn/must-gather-9dcdm" event={"ID":"2544e232-41b9-4494-bdd8-74a0a0d88a59","Type":"ContainerStarted","Data":"6d4606329768d03327e3d2219ea4d3c240a1d98572be5ea85337d7f49bde24f1"}
May 20 15:53:42.526553 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:42.526492    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-2htxn/must-gather-9dcdm" podStartSLOduration=1.700655492 podStartE2EDuration="2.526471744s" podCreationTimestamp="2026-05-20 15:53:40 +0000 UTC" firstStartedPulling="2026-05-20 15:53:41.067524266 +0000 UTC m=+1309.897620165" lastFinishedPulling="2026-05-20 15:53:41.893340518 +0000 UTC m=+1310.723436417" observedRunningTime="2026-05-20 15:53:42.525876254 +0000 UTC m=+1311.355972175" watchObservedRunningTime="2026-05-20 15:53:42.526471744 +0000 UTC m=+1311.356567669"
May 20 15:53:43.449039 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:43.449003    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qs4db_bcacd4aa-c77b-4759-95d6-223503059be9/global-pull-secret-syncer/0.log"
May 20 15:53:43.502141 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:43.502107    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2gh5q_1f0cd5cb-80d8-46cd-bd4f-bd941cb1d78c/konnectivity-agent/0.log"
May 20 15:53:43.633084 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:43.633047    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-154.ec2.internal_f2511ff5939997c60f4a09153c60ac7a/haproxy/0.log"
May 20 15:53:44.819670 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:44.819629    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-8kbs2/must-gather-zgf5k"]
May 20 15:53:44.820644 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:44.820589    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" containerName="copy" containerID="cri-o://e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4" gracePeriod=2
May 20 15:53:44.822882 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:44.822845    2570 status_manager.go:895] "Failed to get status for pod" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" err="pods \"must-gather-zgf5k\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8kbs2\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object"
May 20 15:53:44.823165 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:44.822943    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-8kbs2/must-gather-zgf5k"]
May 20 15:53:45.215675 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.215574    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8kbs2_must-gather-zgf5k_44b94679-1ef8-454f-a3bc-61bcc8ecb777/copy/0.log"
May 20 15:53:45.216492 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.216218    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:45.218100 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.218058    2570 status_manager.go:895] "Failed to get status for pod" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" err="pods \"must-gather-zgf5k\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8kbs2\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object"
May 20 15:53:45.381616 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.381562    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6xqr\" (UniqueName: \"kubernetes.io/projected/44b94679-1ef8-454f-a3bc-61bcc8ecb777-kube-api-access-n6xqr\") pod \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\" (UID: \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\") "
May 20 15:53:45.381985 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.381953    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44b94679-1ef8-454f-a3bc-61bcc8ecb777-must-gather-output\") pod \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\" (UID: \"44b94679-1ef8-454f-a3bc-61bcc8ecb777\") "
May 20 15:53:45.384789 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.383412    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b94679-1ef8-454f-a3bc-61bcc8ecb777-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "44b94679-1ef8-454f-a3bc-61bcc8ecb777" (UID: "44b94679-1ef8-454f-a3bc-61bcc8ecb777"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:53:45.387030 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.386984    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b94679-1ef8-454f-a3bc-61bcc8ecb777-kube-api-access-n6xqr" (OuterVolumeSpecName: "kube-api-access-n6xqr") pod "44b94679-1ef8-454f-a3bc-61bcc8ecb777" (UID: "44b94679-1ef8-454f-a3bc-61bcc8ecb777"). InnerVolumeSpecName "kube-api-access-n6xqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:53:45.387341 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.387310    2570 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44b94679-1ef8-454f-a3bc-61bcc8ecb777-must-gather-output\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:53:45.488979 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.488880    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6xqr\" (UniqueName: \"kubernetes.io/projected/44b94679-1ef8-454f-a3bc-61bcc8ecb777-kube-api-access-n6xqr\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
May 20 15:53:45.523638 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.523559    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8kbs2_must-gather-zgf5k_44b94679-1ef8-454f-a3bc-61bcc8ecb777/copy/0.log"
May 20 15:53:45.527397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.524163    2570 generic.go:358] "Generic (PLEG): container finished" podID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" containerID="e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4" exitCode=143
May 20 15:53:45.527397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.524290    2570 scope.go:117] "RemoveContainer" containerID="e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4"
May 20 15:53:45.527397 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.524445    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kbs2/must-gather-zgf5k"
May 20 15:53:45.537241 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.537192    2570 status_manager.go:895] "Failed to get status for pod" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" err="pods \"must-gather-zgf5k\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8kbs2\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object"
May 20 15:53:45.539597 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.539520    2570 status_manager.go:895] "Failed to get status for pod" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" pod="openshift-must-gather-8kbs2/must-gather-zgf5k" err="pods \"must-gather-zgf5k\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8kbs2\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object"
May 20 15:53:45.553003 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.552652    2570 scope.go:117] "RemoveContainer" containerID="18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21"
May 20 15:53:45.602251 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.601876    2570 scope.go:117] "RemoveContainer" containerID="e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4"
May 20 15:53:45.602952 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:53:45.602622    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4\": container with ID starting with e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4 not found: ID does not exist" containerID="e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4"
May 20 15:53:45.602952 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.602663    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4"} err="failed to get container status \"e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4\": rpc error: code = NotFound desc = could not find container \"e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4\": container with ID starting with e90cb182fb7e8768d1ac056c1637c0e80658c97fac9583e1a69ab45e959f3ed4 not found: ID does not exist"
May 20 15:53:45.602952 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.602689    2570 scope.go:117] "RemoveContainer" containerID="18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21"
May 20 15:53:45.603433 ip-10-0-140-154 kubenswrapper[2570]: E0520 15:53:45.603312    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21\": container with ID starting with 18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21 not found: ID does not exist" containerID="18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21"
May 20 15:53:45.603433 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.603339    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21"} err="failed to get container status \"18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21\": rpc error: code = NotFound desc = could not find container \"18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21\": container with ID starting with 18353ff081463805a00c316a0cff35422c3c1e3ad40229065bb4e3e2927efe21 not found: ID does not exist"
May 20 15:53:45.763685 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:45.763582    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" path="/var/lib/kubelet/pods/44b94679-1ef8-454f-a3bc-61bcc8ecb777/volumes"
May 20 15:53:47.369404 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.369359    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15b3f30b-8c95-4abc-bdb0-7adac4da8d44/alertmanager/0.log"
May 20 15:53:47.404775 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.404728    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15b3f30b-8c95-4abc-bdb0-7adac4da8d44/config-reloader/0.log"
May 20 15:53:47.437911 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.437878    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15b3f30b-8c95-4abc-bdb0-7adac4da8d44/kube-rbac-proxy-web/0.log"
May 20 15:53:47.475525 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.475492    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15b3f30b-8c95-4abc-bdb0-7adac4da8d44/kube-rbac-proxy/0.log"
May 20 15:53:47.514640 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.514605    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15b3f30b-8c95-4abc-bdb0-7adac4da8d44/kube-rbac-proxy-metric/0.log"
May 20 15:53:47.540789 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.540759    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15b3f30b-8c95-4abc-bdb0-7adac4da8d44/prom-label-proxy/0.log"
May 20 15:53:47.565358 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.565312    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15b3f30b-8c95-4abc-bdb0-7adac4da8d44/init-config-reloader/0.log"
May 20 15:53:47.637442 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.637338    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-bch6c_3394f65c-c209-47f9-9af5-15b7e2db5592/kube-state-metrics/0.log"
May 20 15:53:47.664961 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.664927    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-bch6c_3394f65c-c209-47f9-9af5-15b7e2db5592/kube-rbac-proxy-main/0.log"
May 20 15:53:47.693189 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.693154    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-bch6c_3394f65c-c209-47f9-9af5-15b7e2db5592/kube-rbac-proxy-self/0.log"
May 20 15:53:47.731451 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.731423    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d6d988987-jx8z4_a612e0d5-cf46-406e-83fe-448ed1d20967/metrics-server/0.log"
May 20 15:53:47.765649 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.765619    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-fhnzt_21208836-9aa6-4fb9-a7e6-8f15cd60fdb3/monitoring-plugin/0.log"
May 20 15:53:47.802897 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.802854    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d4vp5_417faefd-8f15-4bd5-9974-563fbc982c35/node-exporter/0.log"
May 20 15:53:47.828167 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.828132    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d4vp5_417faefd-8f15-4bd5-9974-563fbc982c35/kube-rbac-proxy/0.log"
May 20 15:53:47.855559 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:47.855525    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d4vp5_417faefd-8f15-4bd5-9974-563fbc982c35/init-textfile/0.log"
May 20 15:53:48.055765 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.055737    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-szwx5_2bddc1d4-016a-4456-b6d7-b2c8ec2a188b/kube-rbac-proxy-main/0.log"
May 20 15:53:48.080380 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.080343    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-szwx5_2bddc1d4-016a-4456-b6d7-b2c8ec2a188b/kube-rbac-proxy-self/0.log"
May 20 15:53:48.104530 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.104497    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-szwx5_2bddc1d4-016a-4456-b6d7-b2c8ec2a188b/openshift-state-metrics/0.log"
May 20 15:53:48.150127 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.150097    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_daa09e04-7ad0-42f0-ac58-c98dd9e63c4c/prometheus/0.log"
May 20 15:53:48.168917 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.168838    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_daa09e04-7ad0-42f0-ac58-c98dd9e63c4c/config-reloader/0.log"
May 20 15:53:48.196205 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.196075    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_daa09e04-7ad0-42f0-ac58-c98dd9e63c4c/thanos-sidecar/0.log"
May 20 15:53:48.223684 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.223647    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_daa09e04-7ad0-42f0-ac58-c98dd9e63c4c/kube-rbac-proxy-web/0.log"
May 20 15:53:48.243190 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.243143    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_daa09e04-7ad0-42f0-ac58-c98dd9e63c4c/kube-rbac-proxy/0.log"
May 20 15:53:48.269856 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.269821    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_daa09e04-7ad0-42f0-ac58-c98dd9e63c4c/kube-rbac-proxy-thanos/0.log"
May 20 15:53:48.297656 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.297631    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_daa09e04-7ad0-42f0-ac58-c98dd9e63c4c/init-config-reloader/0.log"
May 20 15:53:48.339455 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.339418    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-6h7s6_e9221be2-2270-403a-bf4d-5584c6301d71/prometheus-operator/0.log"
May 20 15:53:48.369278 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.369244    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-6h7s6_e9221be2-2270-403a-bf4d-5584c6301d71/kube-rbac-proxy/0.log"
May 20 15:53:48.408007 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.407944    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-5sdbd_f9e81865-bde8-47dd-8c27-3c34161de4bf/prometheus-operator-admission-webhook/0.log"
May 20 15:53:48.458156 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.458072    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55c5c6c85c-7qrft_f6274f8d-c395-4e58-9b3c-511f763a4869/telemeter-client/0.log"
May 20 15:53:48.496391 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.496342    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55c5c6c85c-7qrft_f6274f8d-c395-4e58-9b3c-511f763a4869/reload/0.log"
May 20 15:53:48.531470 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.531438    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55c5c6c85c-7qrft_f6274f8d-c395-4e58-9b3c-511f763a4869/kube-rbac-proxy/0.log"
May 20 15:53:48.575707 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.575671    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797ff79757-tvm9r_48466168-cf44-4695-acea-19a1139b9426/thanos-query/0.log"
May 20 15:53:48.604257 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.604216    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797ff79757-tvm9r_48466168-cf44-4695-acea-19a1139b9426/kube-rbac-proxy-web/0.log"
May 20 15:53:48.640247 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.640212    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797ff79757-tvm9r_48466168-cf44-4695-acea-19a1139b9426/kube-rbac-proxy/0.log"
May 20 15:53:48.673473 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.673444    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797ff79757-tvm9r_48466168-cf44-4695-acea-19a1139b9426/prom-label-proxy/0.log"
May 20 15:53:48.700030 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.700000    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797ff79757-tvm9r_48466168-cf44-4695-acea-19a1139b9426/kube-rbac-proxy-rules/0.log"
May 20 15:53:48.725677 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:48.725649    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797ff79757-tvm9r_48466168-cf44-4695-acea-19a1139b9426/kube-rbac-proxy-metrics/0.log"
May 20 15:53:50.544499 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.544462    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"]
May 20 15:53:50.545094 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.545026    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" containerName="gather"
May 20 15:53:50.545094 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.545052    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" containerName="gather"
May 20 15:53:50.545094 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.545077    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" containerName="copy"
May 20 15:53:50.545094 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.545084    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" containerName="copy"
May 20 15:53:50.545307 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.545203    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" containerName="gather"
May 20 15:53:50.545307 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.545219    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="44b94679-1ef8-454f-a3bc-61bcc8ecb777" containerName="copy"
May 20 15:53:50.551434 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.551405    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.557412 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.557351    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"]
May 20 15:53:50.613184 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.613154    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-697d8c7bb4-wbrl2_84f757ad-1705-4d2c-a2a8-8f419de18ddf/console/0.log"
May 20 15:53:50.642404 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.642322    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-podres\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.642404 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.642394    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k2s2\" (UniqueName: \"kubernetes.io/projected/0827bf05-7c2c-4a76-9846-2f7419d65e0c-kube-api-access-8k2s2\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.642661 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.642441    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-sys\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.642661 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.642492    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-lib-modules\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.642661 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.642594    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-proc\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.743332 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.743296    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-podres\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.743539 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.743344    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8k2s2\" (UniqueName: \"kubernetes.io/projected/0827bf05-7c2c-4a76-9846-2f7419d65e0c-kube-api-access-8k2s2\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.743539 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.743401    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-sys\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.743539 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.743442    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-lib-modules\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.743539 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.743493    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-podres\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.743539 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.743522    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-proc\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.743790 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.743550    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-sys\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.743790 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.743592    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-lib-modules\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.743790 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.743636    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0827bf05-7c2c-4a76-9846-2f7419d65e0c-proc\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.751821 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.751782    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k2s2\" (UniqueName: \"kubernetes.io/projected/0827bf05-7c2c-4a76-9846-2f7419d65e0c-kube-api-access-8k2s2\") pod \"perf-node-gather-daemonset-cp9cz\" (UID: \"0827bf05-7c2c-4a76-9846-2f7419d65e0c\") " pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:50.864696 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:50.864599    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:51.033491 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:51.033040    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"]
May 20 15:53:51.036225 ip-10-0-140-154 kubenswrapper[2570]: W0520 15:53:51.036183    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0827bf05_7c2c_4a76_9846_2f7419d65e0c.slice/crio-e7fb697775f42fb72c00f3176e6f98f2cdab9ced60130c9e3585ca86a8cae41d WatchSource:0}: Error finding container e7fb697775f42fb72c00f3176e6f98f2cdab9ced60130c9e3585ca86a8cae41d: Status 404 returned error can't find the container with id e7fb697775f42fb72c00f3176e6f98f2cdab9ced60130c9e3585ca86a8cae41d
May 20 15:53:51.553779 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:51.553737    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz" event={"ID":"0827bf05-7c2c-4a76-9846-2f7419d65e0c","Type":"ContainerStarted","Data":"680409c9886b8e9fb28bcf9feca74b8d6275be68cd530b6655a91823b3f51492"}
May 20 15:53:51.553779 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:51.553781    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz" event={"ID":"0827bf05-7c2c-4a76-9846-2f7419d65e0c","Type":"ContainerStarted","Data":"e7fb697775f42fb72c00f3176e6f98f2cdab9ced60130c9e3585ca86a8cae41d"}
May 20 15:53:51.554274 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:51.553814    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:53:51.568921 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:51.568853    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz" podStartSLOduration=1.5688351809999999 podStartE2EDuration="1.568835181s" podCreationTimestamp="2026-05-20 15:53:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:53:51.568246889 +0000 UTC m=+1320.398342811" watchObservedRunningTime="2026-05-20 15:53:51.568835181 +0000 UTC m=+1320.398931103"
May 20 15:53:51.851535 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:51.851508    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v6dcb_5f09d029-10af-4b5f-b7b4-533f16160db1/dns/0.log"
May 20 15:53:51.874037 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:51.874007    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v6dcb_5f09d029-10af-4b5f-b7b4-533f16160db1/kube-rbac-proxy/0.log"
May 20 15:53:51.944233 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:51.944203    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2mr9z_ed665954-8c00-4145-b7de-3605ca754aac/dns-node-resolver/0.log"
May 20 15:53:52.454880 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:52.454848    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-994td_1e841a1f-f858-4bb3-8884-7fc59fe8858b/node-ca/0.log"
May 20 15:53:53.579356 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:53.579327    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9t4sb_2f26dc20-4712-42d6-8fd1-a60eca4d0f20/serve-healthcheck-canary/0.log"
May 20 15:53:54.059001 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:54.058971    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kx27r_fd53f4ec-df9b-4e18-85c3-364ad87d120b/kube-rbac-proxy/0.log"
May 20 15:53:54.084492 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:54.084467    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kx27r_fd53f4ec-df9b-4e18-85c3-364ad87d120b/exporter/0.log"
May 20 15:53:54.107123 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:54.107091    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kx27r_fd53f4ec-df9b-4e18-85c3-364ad87d120b/extractor/0.log"
May 20 15:53:56.211826 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:56.211796    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-xdjm4_76a36312-7a13-48b2-919e-4ccc398882a0/server/0.log"
May 20 15:53:56.358393 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:56.358337    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-shczj_9b9981b2-3a71-401f-b54b-36bea294fd25/manager/0.log"
May 20 15:53:56.385398 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:56.385351    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-fbbbs_0f9358a9-7df4-4a83-b1ac-420fbaa4231f/s3-init/0.log"
May 20 15:53:57.567030 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:53:57.567000    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2htxn/perf-node-gather-daemonset-cp9cz"
May 20 15:54:02.092280 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.092225    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2b4js_01c8fe5f-0c58-4e0b-b213-d815ec0448bd/kube-multus-additional-cni-plugins/0.log"
May 20 15:54:02.118672 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.118644    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2b4js_01c8fe5f-0c58-4e0b-b213-d815ec0448bd/egress-router-binary-copy/0.log"
May 20 15:54:02.146183 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.146154    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2b4js_01c8fe5f-0c58-4e0b-b213-d815ec0448bd/cni-plugins/0.log"
May 20 15:54:02.172391 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.172340    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2b4js_01c8fe5f-0c58-4e0b-b213-d815ec0448bd/bond-cni-plugin/0.log"
May 20 15:54:02.200131 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.200101    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2b4js_01c8fe5f-0c58-4e0b-b213-d815ec0448bd/routeoverride-cni/0.log"
May 20 15:54:02.224579 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.224539    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2b4js_01c8fe5f-0c58-4e0b-b213-d815ec0448bd/whereabouts-cni-bincopy/0.log"
May 20 15:54:02.249555 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.249529    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2b4js_01c8fe5f-0c58-4e0b-b213-d815ec0448bd/whereabouts-cni/0.log"
May 20 15:54:02.660408 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.660357    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zrst8_36d114bd-013f-4b9e-a9cc-b201bc41039b/kube-multus/0.log"
May 20 15:54:02.690786 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.690761    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6fh2l_06532912-05d4-4680-bf74-819d7d4a36a0/network-metrics-daemon/0.log"
May 20 15:54:02.711953 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:02.711918    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6fh2l_06532912-05d4-4680-bf74-819d7d4a36a0/kube-rbac-proxy/0.log"
May 20 15:54:03.595852 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:03.595824    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-controller/0.log"
May 20 15:54:03.614505 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:03.614475    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/0.log"
May 20 15:54:03.626268 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:03.626241    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovn-acl-logging/1.log"
May 20 15:54:03.649357 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:03.649331    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/kube-rbac-proxy-node/0.log"
May 20 15:54:03.673260 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:03.673225    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/kube-rbac-proxy-ovn-metrics/0.log"
May 20 15:54:03.695714 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:03.695688    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/northd/0.log"
May 20 15:54:03.722090 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:03.722053    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/nbdb/0.log"
May 20 15:54:03.745908 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:03.745882    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/sbdb/0.log"
May 20 15:54:03.963290 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:03.963212    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw6k2_cfeb4d5d-7e60-4d62-b083-cc887b4163ea/ovnkube-controller/0.log"
May 20 15:54:05.621448 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:05.621354    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-nlb7f_f946a720-3300-4953-a930-3f99f4b9443c/network-check-target-container/0.log"
May 20 15:54:06.687887 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:06.687856    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2bmz8_11ed634e-dd4e-4349-aca3-9374d531eaa8/iptables-alerter/0.log"
May 20 15:54:07.740144 ip-10-0-140-154 kubenswrapper[2570]: I0520 15:54:07.740111    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-nf5mx_07b55da0-8f81-4c9f-b838-877bffe19092/tuned/0.log"