May 26 15:58:33.492842 ip-10-0-129-89 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 26 15:58:33.492853 ip-10-0-129-89 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 26 15:58:33.492859 ip-10-0-129-89 systemd[1]: kubelet.service: Failed with result 'resources'.
May 26 15:58:33.493063 ip-10-0-129-89 systemd[1]: Failed to start Kubernetes Kubelet.
May 26 15:58:43.742868 ip-10-0-129-89 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 26 15:58:43.742888 ip-10-0-129-89 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9a95a3672f274811ad99adc0f6a4ef1a --
May 26 16:00:48.485594 ip-10-0-129-89 systemd[1]: Starting Kubernetes Kubelet...
May 26 16:00:48.999871 ip-10-0-129-89 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:48.999871 ip-10-0-129-89 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 16:00:48.999871 ip-10-0-129-89 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:48.999871 ip-10-0-129-89 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 16:00:48.999871 ip-10-0-129-89 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:49.001632 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.001491    2570 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 16:00:49.008034 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008020    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:49.008034 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008036    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008040    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008043    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008047    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008050    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008053    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008056    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008059    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008062    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008065    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008067    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008070    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008072    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008076    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008078    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008081    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008083    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008086    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008089    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008091    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:49.008100 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008095    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008099    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008102    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008109    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008112    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008115    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008117    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008120    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008123    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008125    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008128    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008130    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008133    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008136    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008138    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008141    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008143    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008146    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008149    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008151    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:49.008585 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008154    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008156    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008159    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008163    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008168    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008171    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008174    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008177    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008179    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008182    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008185    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008189    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008191    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008194    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008197    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008200    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008203    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008205    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:49.009078 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008208    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008210    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008213    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008216    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008219    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008222    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008225    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008227    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008230    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008232    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008235    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008238    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008240    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008243    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008245    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008248    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008250    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008252    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008255    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008258    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:49.009629 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008261    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008263    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008266    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008269    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008272    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008274    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.008290    2570 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009519    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009527    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009530    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009533    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009536    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009539    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009542    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009546    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009548    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009551    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009554    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009556    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009559    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:49.010121 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009562    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009565    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009567    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009570    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009573    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009575    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009578    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009580    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009583    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009585    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009588    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009591    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009594    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009597    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009600    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009602    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009605    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009607    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009611    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009614    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:49.010618 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009617    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009621    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009625    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009628    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009631    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009633    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009636    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009639    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009641    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009644    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009647    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009650    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009653    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009656    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009660    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009662    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009665    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009668    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009670    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:49.011124 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009673    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009675    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009678    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009681    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009683    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009686    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009689    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009693    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009696    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009699    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009702    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009705    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009710    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009713    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009716    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009719    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009721    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009724    2570 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009726    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009729    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:49.011609 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009731    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009734    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009736    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009739    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009742    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009745    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009748    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009750    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009753    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009755    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009758    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009761    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009763    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.009766    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009843    2570 flags.go:64] FLAG: --address="0.0.0.0"
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009850    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009857    2570 flags.go:64] FLAG: --anonymous-auth="true"
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009861    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009866    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009869    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009873    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 16:00:49.012102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009877    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009880    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009884    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009888    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009892    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009895    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009898    2570 flags.go:64] FLAG: --cgroup-root=""
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009901    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009904    2570 flags.go:64] FLAG: --client-ca-file=""
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009907    2570 flags.go:64] FLAG: --cloud-config=""
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009910    2570 flags.go:64] FLAG: --cloud-provider="external"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009912    2570 flags.go:64] FLAG: --cluster-dns="[]"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009916    2570 flags.go:64] FLAG: --cluster-domain=""
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009919    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009922    2570 flags.go:64] FLAG: --config-dir=""
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009925    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009928    2570 flags.go:64] FLAG: --container-log-max-files="5"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009932    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009936    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009939    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009942    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009945    2570 flags.go:64] FLAG: --contention-profiling="false"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009953    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009956    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009959    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 16:00:49.012656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009962    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009966    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009969    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009972    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009975    2570 flags.go:64] FLAG: --enable-load-reader="false"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009978    2570 flags.go:64] FLAG: --enable-server="true"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009981    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009986    2570 flags.go:64] FLAG: --event-burst="100"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009989    2570 flags.go:64] FLAG: --event-qps="50"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009992    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009995    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.009998    2570 flags.go:64] FLAG: --eviction-hard=""
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010002    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010005    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010008    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010011    2570 flags.go:64] FLAG: --eviction-soft=""
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010014    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010017    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010020    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010023    2570 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010026    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010029    2570 flags.go:64] FLAG: --fail-swap-on="true"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010031    2570 flags.go:64] FLAG: --feature-gates=""
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010035    2570 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010038    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 16:00:49.013289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010041    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010045    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010048    2570 flags.go:64] FLAG: --healthz-port="10248"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010051    2570 flags.go:64] FLAG: --help="false"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010055    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-129-89.ec2.internal"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010058    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010061    2570 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010063    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010067    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010070    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010073    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010076    2570 flags.go:64] FLAG: --image-service-endpoint=""
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010078    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010081    2570 flags.go:64] FLAG: --kube-api-burst="100"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010084    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010088    2570 flags.go:64] FLAG: --kube-api-qps="50"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010091    2570 flags.go:64] FLAG: --kube-reserved=""
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010094    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010096    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010099    2570 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010103    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010106    2570 flags.go:64] FLAG: --lock-file=""
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010109    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010111    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 16:00:49.013948 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010115    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010120    2570 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010123    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010126    2570 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010129    2570 flags.go:64] FLAG: --logging-format="text"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010132    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010136    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010139    2570 flags.go:64] FLAG: --manifest-url=""
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010142    2570 flags.go:64] FLAG: --manifest-url-header=""
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010146    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010149    2570 flags.go:64] FLAG: --max-open-files="1000000"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010153    2570 flags.go:64] FLAG: --max-pods="110"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010157    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010160    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010163    2570 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010166    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010169    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010171    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010174    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010182    2570 flags.go:64] FLAG: --node-status-max-images="50"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010185    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010187    2570 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010191    2570 flags.go:64] FLAG: --pod-cidr=""
May 26 16:00:49.014537 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010194    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010199    2570 flags.go:64] FLAG: --pod-manifest-path=""
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010202    2570 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010205    2570 flags.go:64] FLAG: --pods-per-core="0"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010208    2570 flags.go:64] FLAG: --port="10250"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010211    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010214    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0472065673fb47234"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010217    2570 flags.go:64] FLAG: --qos-reserved=""
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010220    2570 flags.go:64] FLAG: --read-only-port="10255"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010223    2570 flags.go:64] FLAG: --register-node="true"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010226    2570 flags.go:64] FLAG: --register-schedulable="true"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010229    2570 flags.go:64] FLAG: --register-with-taints=""
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010232    2570 flags.go:64] FLAG: --registry-burst="10"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010235    2570 flags.go:64] FLAG: --registry-qps="5"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010238    2570 flags.go:64] FLAG: --reserved-cpus=""
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010241    2570 flags.go:64] FLAG: --reserved-memory=""
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010247    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010250    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010253    2570 flags.go:64] FLAG: --rotate-certificates="false"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010256    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010258    2570 flags.go:64] FLAG: --runonce="false"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010261    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010266    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010269    2570 flags.go:64] FLAG: --seccomp-default="false"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010273    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010292    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 16:00:49.015096 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010296    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010299    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010302    2570 flags.go:64] FLAG: --storage-driver-password="root"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010305    2570 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010308    2570 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010311    2570 flags.go:64] FLAG: --storage-driver-user="root"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010314    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010317    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010320    2570 flags.go:64] FLAG: --system-cgroups=""
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010323    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010328    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010331    2570 flags.go:64] FLAG: --tls-cert-file=""
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010334    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010338    2570 flags.go:64] FLAG: --tls-min-version=""
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010341    2570 flags.go:64] FLAG: --tls-private-key-file=""
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010344    2570 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010347    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010350    2570 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010353    2570 flags.go:64] FLAG: --v="2"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010357    2570 flags.go:64] FLAG: --version="false"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010361    2570 flags.go:64] FLAG: --vmodule=""
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010365    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.010369    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010466    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:49.015773 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010470    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010473    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010476    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010479    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010483    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010485    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010490    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010493    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010495    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010499    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010503    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010506    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010509    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010511    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010514    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010516    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010519    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010522    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010525    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010527    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:49.016406 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010529    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010532    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010535    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010538    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010540    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010543    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010546    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010548    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010550    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010553    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010555    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010558    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010561    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010563    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010566    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010568    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010572    2570 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010575    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010578    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:49.017258 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010581    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010584    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010586    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010589    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010592    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010594    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010597    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010599    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010602    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010605    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010607    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010610    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010613    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010615    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010618    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010621    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010624    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010627    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010629    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010632    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:49.018062 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010634    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010637    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010639    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010642    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010644    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010647    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010649    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010652    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010656    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010660    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010663    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010667    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010670    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010673    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010675    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010678    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010681    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010683    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010686    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:49.018677 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010688    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:49.019156 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010691    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:49.019156 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010693    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:49.019156 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010696    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:49.019156 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010698    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:49.019156 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010701    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:49.019156 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.010703    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:49.019156 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.011401    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:49.019510 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.019488    2570 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 16:00:49.019543 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.019511    2570 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 16:00:49.019577 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019562    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:49.019577 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019567    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:49.019577 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019570    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:49.019577 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019574    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:49.019577 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019577    2570 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:49.019577 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019580    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019584    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019586    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019589    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019592    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019594    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019597    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019599    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019602    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019605    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019607    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019610    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019612    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019615    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019617    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019620    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019622    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019625    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019628    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019631    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:49.019766 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019634    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019636    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019639    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019641    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019644    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019646    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019649    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019651    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019654    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019656    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019659    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019662    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019666    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019669    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019674    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019677    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019680    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019683    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019686    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019689    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:49.020253 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019691    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019694    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019697    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019700    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019703    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019705    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019708    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019711    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019713    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019715    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019718    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019721    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019723    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019726    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019728    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019731    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019734    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019736    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019739    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019741    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:49.020767 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019744    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019746    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019749    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019752    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019754    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019757    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019760    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019763    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019766    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019769    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019771    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019775    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019778    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019781    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019784    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019786    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019790    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019794    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019797    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019799    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:49.021244 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019801    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.019807    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019898    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019902    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019904    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019907    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019910    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019913    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019916    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019919    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019921    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019924    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019927    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019930    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019932    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019935    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:49.021763 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019937    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019939    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019943    2570 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019945    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019948    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019951    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019953    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019956    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019959    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019962    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019964    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019967    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019969    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019972    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019974    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019977    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019988    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019991    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019993    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019996    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:49.022158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.019999    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020001    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020004    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020006    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020009    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020011    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020014    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020017    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020020    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020022    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020025    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020028    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020030    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020033    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020036    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020038    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020041    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020044    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020046    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020049    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:49.022669 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020051    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020055    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020057    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020060    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020063    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020065    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020068    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020070    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020073    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020075    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020078    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020080    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020082    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020085    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020088    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020090    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020093    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020095    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020098    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:49.023160 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020101    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020105    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020109    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020112    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020115    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020118    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020121    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020124    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020126    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020129    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020131    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020134    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:49.020136    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.020141    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:49.023656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.020913    2570 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 16:00:49.024014 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.023061    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 16:00:49.024054 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.024042    2570 server.go:1019] "Starting client certificate rotation"
May 26 16:00:49.024156 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.024143    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:00:49.024185 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.024179    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:00:49.052400 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.052382    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:00:49.054297 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.054270    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:00:49.070875 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.070861    2570 log.go:25] "Validated CRI v1 runtime API"
May 26 16:00:49.077238 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.077221    2570 log.go:25] "Validated CRI v1 image API"
May 26 16:00:49.078852 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.078827    2570 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 16:00:49.081641 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.081617    2570 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:00:49.082037 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.082018    2570 fs.go:135] Filesystem UUIDs: map[0605d65d-4c79-4132-acdb-de016b9945a7:/dev/nvme0n1p3 1f95cda0-3d34-40a8-9a57-61f92d2b2128:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
May 26 16:00:49.082078 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.082038    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 16:00:49.086490 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.086381    2570 manager.go:217] Machine: {Timestamp:2026-05-26 16:00:49.085892517 +0000 UTC m=+0.467214533 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108006 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2d3e7aad9ae4085564d5de9735d5 SystemUUID:ec2b2d3e-7aad-9ae4-0855-64d5de9735d5 BootID:9a95a367-2f27-4811-ad99-adc0f6a4ef1a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dd:ec:01:d8:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dd:ec:01:d8:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:5a:bd:53:15:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 16:00:49.086490 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.086485    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 16:00:49.086609 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.086555    2570 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 16:00:49.091416 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.091391    2570 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 16:00:49.091567 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.091419    2570 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 16:00:49.091613 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.091577    2570 topology_manager.go:138] "Creating topology manager with none policy"
May 26 16:00:49.091613 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.091586    2570 container_manager_linux.go:306] "Creating device plugin manager"
May 26 16:00:49.091613 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.091593    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:00:49.092971 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.092960    2570 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:00:49.095340 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.095329    2570 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:00:49.095438 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.095430    2570 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 16:00:49.098356 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.098340    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hw9c7"
May 26 16:00:49.099691 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.099680    2570 kubelet.go:491] "Attempting to sync node with API server"
May 26 16:00:49.099730 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.099701    2570 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 16:00:49.099730 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.099713    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 16:00:49.099730 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.099722    2570 kubelet.go:397] "Adding apiserver pod source"
May 26 16:00:49.099730 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.099731    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 16:00:49.100815 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.100799    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:00:49.100815 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.100817    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:00:49.103485 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.103470    2570 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 16:00:49.105289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.105261    2570 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 16:00:49.105756 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.105743    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hw9c7"
May 26 16:00:49.106332 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106319    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 16:00:49.106399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106336    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 16:00:49.106399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106343    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 16:00:49.106399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106349    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 16:00:49.106399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106357    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 16:00:49.106399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106365    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 16:00:49.106399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106374    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 16:00:49.106399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106388    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 16:00:49.106399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106395    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 16:00:49.106399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106400    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 16:00:49.106707 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106414    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 16:00:49.106707 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.106646    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 16:00:49.107544 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.107530    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 16:00:49.107544 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.107545    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 16:00:49.110724 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.110710    2570 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 16:00:49.110802 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.110745    2570 server.go:1295] "Started kubelet"
May 26 16:00:49.110858 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.110830    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 16:00:49.110906 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.110829    2570 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 16:00:49.110906 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.110889    2570 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 16:00:49.111625 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.111598    2570 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:49.111798 ip-10-0-129-89 systemd[1]: Started Kubernetes Kubelet.
May 26 16:00:49.112029 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.112013    2570 server.go:317] "Adding debug handlers to kubelet server"
May 26 16:00:49.112899 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.112880    2570 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 16:00:49.121209 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.121186    2570 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:49.123264 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.123245    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 16:00:49.123745 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.123727    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 16:00:49.123971 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.123949    2570 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 16:00:49.123971 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.123973    2570 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 16:00:49.124089 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.124017    2570 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 16:00:49.124089 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.124064    2570 reconstruct.go:97] "Volume reconstruction finished"
May 26 16:00:49.124089 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.124073    2570 reconciler.go:26] "Reconciler: start to sync state"
May 26 16:00:49.124670 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.124644    2570 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-89.ec2.internal" not found
May 26 16:00:49.125154 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.125135    2570 factory.go:55] Registering systemd factory
May 26 16:00:49.125154 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.125153    2570 factory.go:223] Registration of the systemd container factory successfully
May 26 16:00:49.125434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.125414    2570 factory.go:153] Registering CRI-O factory
May 26 16:00:49.125502 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.125436    2570 factory.go:223] Registration of the crio container factory successfully
May 26 16:00:49.125502 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.125490    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 16:00:49.125584 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.125508    2570 factory.go:103] Registering Raw factory
May 26 16:00:49.125584 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.125525    2570 manager.go:1196] Started watching for new ooms in manager
May 26 16:00:49.126097 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:49.126070    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-89.ec2.internal\" not found"
May 26 16:00:49.126360 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.126346    2570 manager.go:319] Starting recovery of all containers
May 26 16:00:49.126718 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:49.126639    2570 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 16:00:49.126932 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.126913    2570 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:49.129933 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:49.129912    2570 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-89.ec2.internal\" not found" node="ip-10-0-129-89.ec2.internal"
May 26 16:00:49.136267 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.136101    2570 manager.go:324] Recovery completed
May 26 16:00:49.140782 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.140767    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:49.140871 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.140855    2570 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-89.ec2.internal" not found
May 26 16:00:49.142543 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.142529    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-89.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:49.142600 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.142554    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-89.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:49.142600 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.142563    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-89.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:49.142943 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.142932    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 16:00:49.142992 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.142945    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 16:00:49.142992 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.142971    2570 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:00:49.145848 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.145831    2570 policy_none.go:49] "None policy: Start"
May 26 16:00:49.145848 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.145847    2570 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 16:00:49.145983 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.145861    2570 state_mem.go:35] "Initializing new in-memory state store"
May 26 16:00:49.195854 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.195841    2570 manager.go:341] "Starting Device Plugin manager"
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:49.195875    2570 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.195884    2570 server.go:85] "Starting device plugin registration server"
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.196117    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.196130    2570 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.196231    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.196331    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.196341    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:49.196870    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:49.196909    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-89.ec2.internal\" not found"
May 26 16:00:49.215767 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.197198    2570 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-89.ec2.internal" not found
May 26 16:00:49.229629 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.229605    2570 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 16:00:49.230821 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.230800    2570 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 16:00:49.230899 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.230852    2570 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 16:00:49.230899 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.230869    2570 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 16:00:49.230899 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.230876    2570 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 16:00:49.231009 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:49.230912    2570 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 16:00:49.233162 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.233141    2570 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:49.297102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.297038    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:49.297986 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.297965    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-89.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:49.298086 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.297991    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-89.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:49.298086 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.298001    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-89.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:49.298086 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.298023    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-89.ec2.internal"
May 26 16:00:49.308505 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.308491    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-89.ec2.internal"
May 26 16:00:49.331926 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.331887    2570 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"]
May 26 16:00:49.334059 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.334041    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.334163 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.334041    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.364906 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.364890    2570 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.369194 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.369182    2570 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.378762 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.378743    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:00:49.378840 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.378747    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:00:49.425952 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.425929    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84135cd3e968802e4af704bc56619b8f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-89.ec2.internal\" (UID: \"84135cd3e968802e4af704bc56619b8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.426050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.425956    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9306cf6090e91ae784717834a4e427b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal\" (UID: \"c9306cf6090e91ae784717834a4e427b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.426050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.425978    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9306cf6090e91ae784717834a4e427b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal\" (UID: \"c9306cf6090e91ae784717834a4e427b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.526775 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.526749    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84135cd3e968802e4af704bc56619b8f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-89.ec2.internal\" (UID: \"84135cd3e968802e4af704bc56619b8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.526775 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.526779    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9306cf6090e91ae784717834a4e427b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal\" (UID: \"c9306cf6090e91ae784717834a4e427b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.526963 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.526795    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9306cf6090e91ae784717834a4e427b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal\" (UID: \"c9306cf6090e91ae784717834a4e427b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.526963 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.526831    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9306cf6090e91ae784717834a4e427b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal\" (UID: \"c9306cf6090e91ae784717834a4e427b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.526963 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.526856    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84135cd3e968802e4af704bc56619b8f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-89.ec2.internal\" (UID: \"84135cd3e968802e4af704bc56619b8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.526963 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.526871    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9306cf6090e91ae784717834a4e427b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal\" (UID: \"c9306cf6090e91ae784717834a4e427b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.680483 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.680383    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal"
May 26 16:00:49.682041 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:49.682024    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"
May 26 16:00:50.023970 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.023897    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 16:00:50.024547 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.024037    2570 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:00:50.024547 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.024065    2570 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:00:50.024547 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.024072    2570 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:00:50.100866 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.100839    2570 apiserver.go:52] "Watching apiserver"
May 26 16:00:50.106375 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.106353    2570 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 16:00:50.107917 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.107896    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 15:55:49 +0000 UTC" deadline="2027-10-27 14:36:24.735634176 +0000 UTC"
May 26 16:00:50.107996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.107917    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12454h35m34.627719529s"
May 26 16:00:50.108510 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.108491    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-2pjqf","openshift-ovn-kubernetes/ovnkube-node-8fv6s","kube-system/konnectivity-agent-kbdlx","openshift-image-registry/node-ca-mm79x","openshift-multus/multus-k7bjv","openshift-multus/network-metrics-daemon-r88dc","openshift-network-diagnostics/network-check-target-968nq","openshift-network-operator/iptables-alerter-27bnz","kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh","openshift-cluster-node-tuning-operator/tuned-62bbh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal"]
May 26 16:00:50.110902 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.110884    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.113111 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.113089    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 16:00:50.113226 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.113150    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 16:00:50.113226 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.113212    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 16:00:50.113370 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.113230    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 16:00:50.113370 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.113322    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 16:00:50.113474 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.113373    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6d4qv\""
May 26 16:00:50.115541 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.115521    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.117738 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.117712    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:00:50.118394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.118365    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 16:00:50.118497 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.118399    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 16:00:50.118556 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.118497    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 16:00:50.118556 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.118503    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 16:00:50.118648 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.118621    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 16:00:50.118696 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.118645    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qz8wz\""
May 26 16:00:50.118962 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.118948    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 16:00:50.119654 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.119626    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 16:00:50.119751 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.119655    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p72cf\""
May 26 16:00:50.119811 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.119768    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 16:00:50.119940 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.119922    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.120099 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.120083    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.121952 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.121931    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 16:00:50.121952 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.121919    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 16:00:50.122094 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.121957    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8tnc4\""
May 26 16:00:50.123065 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.122703    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 16:00:50.123065 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.122991    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 16:00:50.123065 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.123029    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kfhvp\""
May 26 16:00:50.123475 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.123454    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 16:00:50.126398 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.126365    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:50.126493 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.126462    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:00:50.126552 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.126496    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:50.126600 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.126576    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:00:50.128648 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128633    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.128748 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128730    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be3af2fc-3d53-46f3-b681-d665b14c11a8-cni-binary-copy\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.128805 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128766    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-run-netns\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.128805 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128793    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-var-lib-cni-multus\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.128916 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128818    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-os-release\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.128916 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128843    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-kubelet\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.128916 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128886    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-etc-openvswitch\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129059 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128920    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-run-openvswitch\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129059 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-cni-dir\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129059 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128972    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-var-lib-kubelet\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129059 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.128994    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-etc-kubernetes\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129059 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129032    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129067    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-slash\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129097    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-var-lib-openvswitch\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129126    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129153    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129184    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-run-multus-certs\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129210    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-run-ovn-kubernetes\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129245    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztb8z\" (UniqueName: \"kubernetes.io/projected/270dc9ab-90fe-4504-8313-236f5a761696-kube-api-access-ztb8z\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129268    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-run-k8s-cni-cncf-io\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129297    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-var-lib-cni-bin\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129312    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-conf-dir\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129326    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-system-cni-dir\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129355    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129384    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-systemd-units\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129409    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-cni-netd\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129434    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/270dc9ab-90fe-4504-8313-236f5a761696-ovn-node-metrics-cert\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129460    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129487    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129525    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-socket-dir-parent\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129565    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-hostroot\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129604    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-daemon-config\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129627    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-log-socket\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129650    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/270dc9ab-90fe-4504-8313-236f5a761696-ovnkube-script-lib\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129676    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-cnibin\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129698    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsqw\" (UniqueName: \"kubernetes.io/projected/be3af2fc-3d53-46f3-b681-d665b14c11a8-kube-api-access-hzsqw\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129724    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-run-systemd\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129747    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/270dc9ab-90fe-4504-8313-236f5a761696-env-overrides\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.129877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129769    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bda7246b-9f6c-4339-86e9-2a97c2370213-serviceca\") pod \"node-ca-mm79x\" (UID: \"bda7246b-9f6c-4339-86e9-2a97c2370213\") " pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129794    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpj7\" (UniqueName: \"kubernetes.io/projected/bda7246b-9f6c-4339-86e9-2a97c2370213-kube-api-access-9bpj7\") pod \"node-ca-mm79x\" (UID: \"bda7246b-9f6c-4339-86e9-2a97c2370213\") " pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129820    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdlp\" (UniqueName: \"kubernetes.io/projected/02034fc7-cc11-48cf-8eff-ae2384a2080e-kube-api-access-bkdlp\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129876    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-system-cni-dir\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129902    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-run-netns\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129924    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-node-log\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129945    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/270dc9ab-90fe-4504-8313-236f5a761696-ovnkube-config\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129966    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9185f8d9-5823-4f14-bfb4-7d311127a7f1-konnectivity-ca\") pod \"konnectivity-agent-kbdlx\" (UID: \"9185f8d9-5823-4f14-bfb4-7d311127a7f1\") " pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.129987    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda7246b-9f6c-4339-86e9-2a97c2370213-host\") pod \"node-ca-mm79x\" (UID: \"bda7246b-9f6c-4339-86e9-2a97c2370213\") " pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130013    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck2b7\" (UniqueName: \"kubernetes.io/projected/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-kube-api-access-ck2b7\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130039    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-os-release\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130060    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-cnibin\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130091    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-run-ovn\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130111    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-cni-bin\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.130466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130132    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9185f8d9-5823-4f14-bfb4-7d311127a7f1-agent-certs\") pod \"konnectivity-agent-kbdlx\" (UID: \"9185f8d9-5823-4f14-bfb4-7d311127a7f1\") " pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:00:50.131050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130492    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 16:00:50.131050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130672    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 16:00:50.131050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130694    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 16:00:50.131050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130718    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dr2pn\""
May 26 16:00:50.131050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.130936    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.133216 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.133194    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.133590 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.133567    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 16:00:50.133726 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.133569    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 16:00:50.133726 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.133653    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h8xtp\""
May 26 16:00:50.133726 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.133666    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 16:00:50.135195 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.135168    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 16:00:50.135382 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.135316    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 16:00:50.135544 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.135527    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k5f4l\""
May 26 16:00:50.136951 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.136933    2570 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:00:50.158390 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.158370    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tk7z7"
May 26 16:00:50.164209 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.163508    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tk7z7"
May 26 16:00:50.224683 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.224665    2570 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 16:00:50.230560 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230532    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be3af2fc-3d53-46f3-b681-d665b14c11a8-cni-binary-copy\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.230643 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230565    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-run-netns\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.230643 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230590    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-kubelet\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.230761 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230653    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-kubelet\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.230761 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230712    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-run-netns\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.230761 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230753    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-etc-openvswitch\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.230911 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230803    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-run-openvswitch\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.230911 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230848    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-etc-openvswitch\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.230911 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230882    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-run-openvswitch\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.231062 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230944    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-kubernetes\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.231062 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.230983    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-cni-dir\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.231062 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231023    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-var-lib-kubelet\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.231062 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231048    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.231255 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231092    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-var-lib-kubelet\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.231620 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231587    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-cni-dir\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.231692 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231452    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be3af2fc-3d53-46f3-b681-d665b14c11a8-cni-binary-copy\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.231777 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231744    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-var-lib-openvswitch\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.231853 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231839    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-etc-selinux\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.231907 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-modprobe-d\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.231957 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p9wk\" (UniqueName: \"kubernetes.io/projected/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-kube-api-access-9p9wk\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.232013 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.231999    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-run-multus-certs\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.232065 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232027    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ztb8z\" (UniqueName: \"kubernetes.io/projected/270dc9ab-90fe-4504-8313-236f5a761696-kube-api-access-ztb8z\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.232065 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232051    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.232162 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232083    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda7246b-9f6c-4339-86e9-2a97c2370213-host\") pod \"node-ca-mm79x\" (UID: \"bda7246b-9f6c-4339-86e9-2a97c2370213\") " pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.232162 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232147    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-var-lib-cni-bin\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.232263 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232176    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-conf-dir\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.232263 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232241    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-system-cni-dir\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.232385 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232327    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.232385 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232371    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-systemd-units\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.232476 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232380    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.232528 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232390    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-cni-netd\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.232528 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232515    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.232673 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232657    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda7246b-9f6c-4339-86e9-2a97c2370213-host\") pod \"node-ca-mm79x\" (UID: \"bda7246b-9f6c-4339-86e9-2a97c2370213\") " pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.232673 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232664    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.232824 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232744    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-var-lib-cni-bin\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.232824 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232795    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-run-multus-certs\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.232932 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232833    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-conf-dir\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.232932 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.232858    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-socket-dir-parent\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.233087 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.233061    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-var-lib-openvswitch\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.233343 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.233231    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-daemon-config\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.233421 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.233358    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-cni-netd\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.233885 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.233859    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-systemd-units\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.233992 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.233970    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-daemon-config\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.234054 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.233986    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-system-cni-dir\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.234054 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.233949    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.234054 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234039    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-multus-socket-dir-parent\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.234246 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234228    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-sys-fs\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.234380 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234362    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-sysconfig\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.234484 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234471    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/270dc9ab-90fe-4504-8313-236f5a761696-env-overrides\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.234578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234564    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpj7\" (UniqueName: \"kubernetes.io/projected/bda7246b-9f6c-4339-86e9-2a97c2370213-kube-api-access-9bpj7\") pod \"node-ca-mm79x\" (UID: \"bda7246b-9f6c-4339-86e9-2a97c2370213\") " pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.234681 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234668    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdlp\" (UniqueName: \"kubernetes.io/projected/02034fc7-cc11-48cf-8eff-ae2384a2080e-kube-api-access-bkdlp\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:50.234780 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234764    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-registration-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.234861 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234850    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-device-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.234955 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234942    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-system-cni-dir\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.235053 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235033    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-system-cni-dir\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.235111 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235039    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-node-log\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.235111 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235098    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-socket-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.235192 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235155    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ck2b7\" (UniqueName: \"kubernetes.io/projected/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-kube-api-access-ck2b7\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.235192 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.234941    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/270dc9ab-90fe-4504-8313-236f5a761696-env-overrides\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.235297 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235191    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-os-release\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.235297 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235240    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-cnibin\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.235297 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235248    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-os-release\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.235435 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235299    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-cni-bin\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.235435 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235353    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-sysctl-d\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.235435 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235410    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-var-lib-cni-multus\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.235578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235446    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-os-release\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.235578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235472    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:50.235578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235500    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-etc-kubernetes\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.235578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235542    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-slash\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.235751 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235598    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:50.235751 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235601    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-var-lib-cni-multus\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.235751 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235646    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9wh\" (UniqueName: \"kubernetes.io/projected/557f56c9-c491-4f35-99de-92a9cdc50f23-kube-api-access-sg9wh\") pod \"iptables-alerter-27bnz\" (UID: \"557f56c9-c491-4f35-99de-92a9cdc50f23\") " pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.235751 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235698    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-run-ovn-kubernetes\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.235751 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.235700    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:50.235751 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235730    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.235996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235783    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.235996 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.235798    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs podName:02034fc7-cc11-48cf-8eff-ae2384a2080e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:50.735747669 +0000 UTC m=+2.117069693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs") pod "network-metrics-daemon-r88dc" (UID: "02034fc7-cc11-48cf-8eff-ae2384a2080e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:50.235996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235822    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkm8f\" (UniqueName: \"kubernetes.io/projected/7e844313-be2a-4801-966e-6a1257eaeef4-kube-api-access-bkm8f\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.235996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235827    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-cnibin\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.235996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235861    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-host\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.235996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235891    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-run-k8s-cni-cncf-io\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.235996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235903    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-os-release\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.235996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235926    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/270dc9ab-90fe-4504-8313-236f5a761696-ovn-node-metrics-cert\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.235996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.235968    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236032    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-hostroot\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236059    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-log-socket\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236128    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/270dc9ab-90fe-4504-8313-236f5a761696-ovnkube-script-lib\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236168    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bda7246b-9f6c-4339-86e9-2a97c2370213-serviceca\") pod \"node-ca-mm79x\" (UID: \"bda7246b-9f6c-4339-86e9-2a97c2370213\") " pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236211    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/557f56c9-c491-4f35-99de-92a9cdc50f23-iptables-alerter-script\") pod \"iptables-alerter-27bnz\" (UID: \"557f56c9-c491-4f35-99de-92a9cdc50f23\") " pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236252    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-cnibin\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236257    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236326    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsqw\" (UniqueName: \"kubernetes.io/projected/be3af2fc-3d53-46f3-b681-d665b14c11a8-kube-api-access-hzsqw\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236364    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-run-systemd\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236386    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-cni-bin\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.236434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236399    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-sysctl-conf\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236477    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-host-run-k8s-cni-cncf-io\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236510    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-cnibin\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236551    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-systemd\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236554    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-etc-kubernetes\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236588    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-tuned\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236612    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-slash\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236625    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-run-netns\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236680    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/270dc9ab-90fe-4504-8313-236f5a761696-ovnkube-config\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236721    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9185f8d9-5823-4f14-bfb4-7d311127a7f1-konnectivity-ca\") pod \"konnectivity-agent-kbdlx\" (UID: \"9185f8d9-5823-4f14-bfb4-7d311127a7f1\") " pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236755    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-run\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236787    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-sys\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236837    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-lib-modules\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.236933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236869    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-var-lib-kubelet\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.237639 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.236901    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/557f56c9-c491-4f35-99de-92a9cdc50f23-host-slash\") pod \"iptables-alerter-27bnz\" (UID: \"557f56c9-c491-4f35-99de-92a9cdc50f23\") " pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.237687 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237640    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-run-ovn-kubernetes\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.237687 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237140    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-host-run-netns\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.237781 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237641    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/270dc9ab-90fe-4504-8313-236f5a761696-ovnkube-script-lib\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.237781 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237707    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-run-ovn\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.237781 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237741    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9185f8d9-5823-4f14-bfb4-7d311127a7f1-agent-certs\") pod \"konnectivity-agent-kbdlx\" (UID: \"9185f8d9-5823-4f14-bfb4-7d311127a7f1\") " pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:00:50.237920 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-tmp\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.237920 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237879    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-run-systemd\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.237920 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237903    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-log-socket\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.237920 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237588    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.238100 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237939    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be3af2fc-3d53-46f3-b681-d665b14c11a8-hostroot\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.238100 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237949    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bda7246b-9f6c-4339-86e9-2a97c2370213-serviceca\") pod \"node-ca-mm79x\" (UID: \"bda7246b-9f6c-4339-86e9-2a97c2370213\") " pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.238100 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237971    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-run-ovn\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.238312 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.237984    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/270dc9ab-90fe-4504-8313-236f5a761696-ovnkube-config\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.238456 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.238421    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/270dc9ab-90fe-4504-8313-236f5a761696-node-log\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.239000 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.238979    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9185f8d9-5823-4f14-bfb4-7d311127a7f1-konnectivity-ca\") pod \"konnectivity-agent-kbdlx\" (UID: \"9185f8d9-5823-4f14-bfb4-7d311127a7f1\") " pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:00:50.239736 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.239707    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/270dc9ab-90fe-4504-8313-236f5a761696-ovn-node-metrics-cert\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.240690 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.240667    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9185f8d9-5823-4f14-bfb4-7d311127a7f1-agent-certs\") pod \"konnectivity-agent-kbdlx\" (UID: \"9185f8d9-5823-4f14-bfb4-7d311127a7f1\") " pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:00:50.241073 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.241053    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztb8z\" (UniqueName: \"kubernetes.io/projected/270dc9ab-90fe-4504-8313-236f5a761696-kube-api-access-ztb8z\") pod \"ovnkube-node-8fv6s\" (UID: \"270dc9ab-90fe-4504-8313-236f5a761696\") " pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.242343 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.242322    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:50.242343 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.242345    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:50.242480 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.242360    2570 projected.go:194] Error preparing data for projected volume kube-api-access-j6n5l for pod openshift-network-diagnostics/network-check-target-968nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:50.242480 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.242418    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l podName:49b37583-51bd-40e5-94c6-e1b191d0a891 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:50.742400993 +0000 UTC m=+2.123722997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j6n5l" (UniqueName: "kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l") pod "network-check-target-968nq" (UID: "49b37583-51bd-40e5-94c6-e1b191d0a891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:50.242969 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.242944    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck2b7\" (UniqueName: \"kubernetes.io/projected/59beaf4f-19bd-4f7c-be73-07dad7f9fde2-kube-api-access-ck2b7\") pod \"multus-additional-cni-plugins-2pjqf\" (UID: \"59beaf4f-19bd-4f7c-be73-07dad7f9fde2\") " pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.243042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.242979    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdlp\" (UniqueName: \"kubernetes.io/projected/02034fc7-cc11-48cf-8eff-ae2384a2080e-kube-api-access-bkdlp\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:50.243042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.242994    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpj7\" (UniqueName: \"kubernetes.io/projected/bda7246b-9f6c-4339-86e9-2a97c2370213-kube-api-access-9bpj7\") pod \"node-ca-mm79x\" (UID: \"bda7246b-9f6c-4339-86e9-2a97c2370213\") " pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.244809 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.244790    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsqw\" (UniqueName: \"kubernetes.io/projected/be3af2fc-3d53-46f3-b681-d665b14c11a8-kube-api-access-hzsqw\") pod \"multus-k7bjv\" (UID: \"be3af2fc-3d53-46f3-b681-d665b14c11a8\") " pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.258511 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.258475    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84135cd3e968802e4af704bc56619b8f.slice/crio-faf0289b133cba685f77c13bd80b2ab4d30e001ae9ea15e9d342e17589b7ca90 WatchSource:0}: Error finding container faf0289b133cba685f77c13bd80b2ab4d30e001ae9ea15e9d342e17589b7ca90: Status 404 returned error can't find the container with id faf0289b133cba685f77c13bd80b2ab4d30e001ae9ea15e9d342e17589b7ca90
May 26 16:00:50.258795 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.258759    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9306cf6090e91ae784717834a4e427b.slice/crio-9f9c69c540f9d3c82c02d804e4876eb3984701a665aa7b9d10314badd6d5d816 WatchSource:0}: Error finding container 9f9c69c540f9d3c82c02d804e4876eb3984701a665aa7b9d10314badd6d5d816: Status 404 returned error can't find the container with id 9f9c69c540f9d3c82c02d804e4876eb3984701a665aa7b9d10314badd6d5d816
May 26 16:00:50.262827 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.262814    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:00:50.338770 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338704    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-sys-fs\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.338770 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338723    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-sys-fs\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.338770 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338745    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-sysconfig\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.338770 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338766    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-registration-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338781    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-device-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338798    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-socket-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338816    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-sysctl-d\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338837    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-device-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338857    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9wh\" (UniqueName: \"kubernetes.io/projected/557f56c9-c491-4f35-99de-92a9cdc50f23-kube-api-access-sg9wh\") pod \"iptables-alerter-27bnz\" (UID: \"557f56c9-c491-4f35-99de-92a9cdc50f23\") " pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338872    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-registration-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338870    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-sysconfig\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338876    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bkm8f\" (UniqueName: \"kubernetes.io/projected/7e844313-be2a-4801-966e-6a1257eaeef4-kube-api-access-bkm8f\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338922    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-host\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338935    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-socket-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.338975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338968    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/557f56c9-c491-4f35-99de-92a9cdc50f23-iptables-alerter-script\") pod \"iptables-alerter-27bnz\" (UID: \"557f56c9-c491-4f35-99de-92a9cdc50f23\") " pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.338980    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-host\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339009    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-sysctl-conf\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339028    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-sysctl-d\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339034    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-systemd\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339073    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-tuned\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339101    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-run\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339119    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-systemd\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339124    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-sys\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339169    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-sys\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339177    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-sysctl-conf\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339192    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-run\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339231    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-lib-modules\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339259    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-var-lib-kubelet\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339301    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/557f56c9-c491-4f35-99de-92a9cdc50f23-host-slash\") pod \"iptables-alerter-27bnz\" (UID: \"557f56c9-c491-4f35-99de-92a9cdc50f23\") " pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339340    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-var-lib-kubelet\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339346    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-tmp\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339378    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-lib-modules\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.339458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339383    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-kubernetes\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339416    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-etc-selinux\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339422    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-kubernetes\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339441    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-modprobe-d\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339467    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9p9wk\" (UniqueName: \"kubernetes.io/projected/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-kube-api-access-9p9wk\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339498    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339515    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-etc-selinux\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339372    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/557f56c9-c491-4f35-99de-92a9cdc50f23-host-slash\") pod \"iptables-alerter-27bnz\" (UID: \"557f56c9-c491-4f35-99de-92a9cdc50f23\") " pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339549    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/557f56c9-c491-4f35-99de-92a9cdc50f23-iptables-alerter-script\") pod \"iptables-alerter-27bnz\" (UID: \"557f56c9-c491-4f35-99de-92a9cdc50f23\") " pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339567    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e844313-be2a-4801-966e-6a1257eaeef4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.340134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.339602    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-modprobe-d\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.341709 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.341674    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-tmp\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.341784 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.341714    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-etc-tuned\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.347081 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.347055    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkm8f\" (UniqueName: \"kubernetes.io/projected/7e844313-be2a-4801-966e-6a1257eaeef4-kube-api-access-bkm8f\") pod \"aws-ebs-csi-driver-node-5cnkh\" (UID: \"7e844313-be2a-4801-966e-6a1257eaeef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.347165 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.347148    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9wh\" (UniqueName: \"kubernetes.io/projected/557f56c9-c491-4f35-99de-92a9cdc50f23-kube-api-access-sg9wh\") pod \"iptables-alerter-27bnz\" (UID: \"557f56c9-c491-4f35-99de-92a9cdc50f23\") " pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.347375 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.347360    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p9wk\" (UniqueName: \"kubernetes.io/projected/ddfb6a13-33b0-4ea4-ac38-ec92b912632e-kube-api-access-9p9wk\") pod \"tuned-62bbh\" (UID: \"ddfb6a13-33b0-4ea4-ac38-ec92b912632e\") " pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.438474 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.438447    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pjqf"
May 26 16:00:50.445237 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.445215    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59beaf4f_19bd_4f7c_be73_07dad7f9fde2.slice/crio-dbdd4ff50fd7cba16d7edd8e49e83d7f0393ea080f240a31dc0a4505cdcdebee WatchSource:0}: Error finding container dbdd4ff50fd7cba16d7edd8e49e83d7f0393ea080f240a31dc0a4505cdcdebee: Status 404 returned error can't find the container with id dbdd4ff50fd7cba16d7edd8e49e83d7f0393ea080f240a31dc0a4505cdcdebee
May 26 16:00:50.459763 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.459747    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:00:50.465263 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.465235    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270dc9ab_90fe_4504_8313_236f5a761696.slice/crio-e9985b62664300b2948aa0ffb7f702cb09e0d8d31e26126e9f7a09432551c0e7 WatchSource:0}: Error finding container e9985b62664300b2948aa0ffb7f702cb09e0d8d31e26126e9f7a09432551c0e7: Status 404 returned error can't find the container with id e9985b62664300b2948aa0ffb7f702cb09e0d8d31e26126e9f7a09432551c0e7
May 26 16:00:50.473630 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.473614    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:00:50.479355 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.479333    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9185f8d9_5823_4f14_bfb4_7d311127a7f1.slice/crio-d8f5859f27e1ba1a79d549f867eced7b3a680752c9c1f2ec5dcd9c25ade44113 WatchSource:0}: Error finding container d8f5859f27e1ba1a79d549f867eced7b3a680752c9c1f2ec5dcd9c25ade44113: Status 404 returned error can't find the container with id d8f5859f27e1ba1a79d549f867eced7b3a680752c9c1f2ec5dcd9c25ade44113
May 26 16:00:50.499799 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.499769    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mm79x"
May 26 16:00:50.506450 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.506430    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda7246b_9f6c_4339_86e9_2a97c2370213.slice/crio-1aec5eec2835f06744664c85a474a8485449ba36036cd510e9db0acce07cc653 WatchSource:0}: Error finding container 1aec5eec2835f06744664c85a474a8485449ba36036cd510e9db0acce07cc653: Status 404 returned error can't find the container with id 1aec5eec2835f06744664c85a474a8485449ba36036cd510e9db0acce07cc653
May 26 16:00:50.519944 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.519928    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7bjv"
May 26 16:00:50.525655 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.525632    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3af2fc_3d53_46f3_b681_d665b14c11a8.slice/crio-f9a958417f9e7ca89c6a70ee675c779f25b335c2f3bf5e69588c8a3f38a8337e WatchSource:0}: Error finding container f9a958417f9e7ca89c6a70ee675c779f25b335c2f3bf5e69588c8a3f38a8337e: Status 404 returned error can't find the container with id f9a958417f9e7ca89c6a70ee675c779f25b335c2f3bf5e69588c8a3f38a8337e
May 26 16:00:50.534877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.534861    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-27bnz"
May 26 16:00:50.539771 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.539749    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557f56c9_c491_4f35_99de_92a9cdc50f23.slice/crio-be727a4b94f2adafb608fc48ca6a26ce8cb90850c07e86929b03f9993428c127 WatchSource:0}: Error finding container be727a4b94f2adafb608fc48ca6a26ce8cb90850c07e86929b03f9993428c127: Status 404 returned error can't find the container with id be727a4b94f2adafb608fc48ca6a26ce8cb90850c07e86929b03f9993428c127
May 26 16:00:50.548606 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.548584    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh"
May 26 16:00:50.554109 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.554089    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e844313_be2a_4801_966e_6a1257eaeef4.slice/crio-090fd3fb461a7d7602f965d1024a0723ab33d8a87391ce8bfccb4a8db236eb1b WatchSource:0}: Error finding container 090fd3fb461a7d7602f965d1024a0723ab33d8a87391ce8bfccb4a8db236eb1b: Status 404 returned error can't find the container with id 090fd3fb461a7d7602f965d1024a0723ab33d8a87391ce8bfccb4a8db236eb1b
May 26 16:00:50.562269 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.562252    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-62bbh"
May 26 16:00:50.567957 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:00:50.567940    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfb6a13_33b0_4ea4_ac38_ec92b912632e.slice/crio-40b1761894423ea232fa46796a26fae9cc70a2aef7110c33b848e2f804f815eb WatchSource:0}: Error finding container 40b1761894423ea232fa46796a26fae9cc70a2aef7110c33b848e2f804f815eb: Status 404 returned error can't find the container with id 40b1761894423ea232fa46796a26fae9cc70a2aef7110c33b848e2f804f815eb
May 26 16:00:50.741788 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.741725    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:50.741946 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.741862    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:50.741946 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.741926    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs podName:02034fc7-cc11-48cf-8eff-ae2384a2080e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:51.741907004 +0000 UTC m=+3.123229015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs") pod "network-metrics-daemon-r88dc" (UID: "02034fc7-cc11-48cf-8eff-ae2384a2080e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:50.842598 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:50.842545    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:50.842749 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.842703    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:50.842749 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.842727    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:50.842860 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.842752    2570 projected.go:194] Error preparing data for projected volume kube-api-access-j6n5l for pod openshift-network-diagnostics/network-check-target-968nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:50.842860 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:50.842819    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l podName:49b37583-51bd-40e5-94c6-e1b191d0a891 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:51.842799528 +0000 UTC m=+3.224121534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6n5l" (UniqueName: "kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l") pod "network-check-target-968nq" (UID: "49b37583-51bd-40e5-94c6-e1b191d0a891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:51.022722 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.022687    2570 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:51.165471 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.165386    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:50 +0000 UTC" deadline="2027-12-22 12:33:07.942423207 +0000 UTC"
May 26 16:00:51.165471 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.165427    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13796h32m16.776999953s"
May 26 16:00:51.191673 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.191479    2570 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:51.250728 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.250593    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal" event={"ID":"84135cd3e968802e4af704bc56619b8f","Type":"ContainerStarted","Data":"faf0289b133cba685f77c13bd80b2ab4d30e001ae9ea15e9d342e17589b7ca90"}
May 26 16:00:51.252098 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.251744    2570 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:51.259186 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.259156    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-27bnz" event={"ID":"557f56c9-c491-4f35-99de-92a9cdc50f23","Type":"ContainerStarted","Data":"be727a4b94f2adafb608fc48ca6a26ce8cb90850c07e86929b03f9993428c127"}
May 26 16:00:51.265258 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.265198    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7bjv" event={"ID":"be3af2fc-3d53-46f3-b681-d665b14c11a8","Type":"ContainerStarted","Data":"f9a958417f9e7ca89c6a70ee675c779f25b335c2f3bf5e69588c8a3f38a8337e"}
May 26 16:00:51.275363 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.275244    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mm79x" event={"ID":"bda7246b-9f6c-4339-86e9-2a97c2370213","Type":"ContainerStarted","Data":"1aec5eec2835f06744664c85a474a8485449ba36036cd510e9db0acce07cc653"}
May 26 16:00:51.286241 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.286097    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kbdlx" event={"ID":"9185f8d9-5823-4f14-bfb4-7d311127a7f1","Type":"ContainerStarted","Data":"d8f5859f27e1ba1a79d549f867eced7b3a680752c9c1f2ec5dcd9c25ade44113"}
May 26 16:00:51.294514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.294469    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" event={"ID":"270dc9ab-90fe-4504-8313-236f5a761696","Type":"ContainerStarted","Data":"e9985b62664300b2948aa0ffb7f702cb09e0d8d31e26126e9f7a09432551c0e7"}
May 26 16:00:51.310341 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.309952    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pjqf" event={"ID":"59beaf4f-19bd-4f7c-be73-07dad7f9fde2","Type":"ContainerStarted","Data":"dbdd4ff50fd7cba16d7edd8e49e83d7f0393ea080f240a31dc0a4505cdcdebee"}
May 26 16:00:51.326610 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.326576    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal" event={"ID":"c9306cf6090e91ae784717834a4e427b","Type":"ContainerStarted","Data":"9f9c69c540f9d3c82c02d804e4876eb3984701a665aa7b9d10314badd6d5d816"}
May 26 16:00:51.338352 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.338299    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-62bbh" event={"ID":"ddfb6a13-33b0-4ea4-ac38-ec92b912632e","Type":"ContainerStarted","Data":"40b1761894423ea232fa46796a26fae9cc70a2aef7110c33b848e2f804f815eb"}
May 26 16:00:51.356770 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.356712    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh" event={"ID":"7e844313-be2a-4801-966e-6a1257eaeef4","Type":"ContainerStarted","Data":"090fd3fb461a7d7602f965d1024a0723ab33d8a87391ce8bfccb4a8db236eb1b"}
May 26 16:00:51.751091 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.750416    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:51.751091 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:51.750588    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:51.751091 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:51.750649    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs podName:02034fc7-cc11-48cf-8eff-ae2384a2080e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:53.75063092 +0000 UTC m=+5.131952927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs") pod "network-metrics-daemon-r88dc" (UID: "02034fc7-cc11-48cf-8eff-ae2384a2080e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:51.851549 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:51.851510    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:51.851732 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:51.851709    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:51.851795 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:51.851742    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:51.851795 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:51.851756    2570 projected.go:194] Error preparing data for projected volume kube-api-access-j6n5l for pod openshift-network-diagnostics/network-check-target-968nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:51.851893 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:51.851813    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l podName:49b37583-51bd-40e5-94c6-e1b191d0a891 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:53.851795207 +0000 UTC m=+5.233117230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6n5l" (UniqueName: "kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l") pod "network-check-target-968nq" (UID: "49b37583-51bd-40e5-94c6-e1b191d0a891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:52.166628 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:52.166540    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:50 +0000 UTC" deadline="2028-02-24 16:30:49.463123602 +0000 UTC"
May 26 16:00:52.166628 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:52.166583    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15336h29m57.296547246s"
May 26 16:00:52.232150 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:52.232120    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:52.232364 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:52.232328    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:00:52.232783 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:52.232760    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:52.232901 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:52.232859    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:00:53.767784 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:53.767709    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:53.768244 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:53.767887    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:53.768244 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:53.767971    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs podName:02034fc7-cc11-48cf-8eff-ae2384a2080e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.767949076 +0000 UTC m=+9.149271083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs") pod "network-metrics-daemon-r88dc" (UID: "02034fc7-cc11-48cf-8eff-ae2384a2080e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:53.868956 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:53.868907    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:53.869139 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:53.869107    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:53.869139 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:53.869128    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:53.869139 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:53.869140    2570 projected.go:194] Error preparing data for projected volume kube-api-access-j6n5l for pod openshift-network-diagnostics/network-check-target-968nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:53.869393 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:53.869196    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l podName:49b37583-51bd-40e5-94c6-e1b191d0a891 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.869177949 +0000 UTC m=+9.250499956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6n5l" (UniqueName: "kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l") pod "network-check-target-968nq" (UID: "49b37583-51bd-40e5-94c6-e1b191d0a891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:54.231945 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:54.231828    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:54.232105 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:54.231959    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:00:54.232411 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:54.232392    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:54.232512 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:54.232493    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:00:55.692538 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.692505    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-qm4f5"]
May 26 16:00:55.696455 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.696425    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:55.700087 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.699531    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hqbdt\""
May 26 16:00:55.700087 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.699810    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 16:00:55.700087 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.699973    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 16:00:55.787555 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.787512    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3f547af-2cf4-4821-ae28-e2087a814658-hosts-file\") pod \"node-resolver-qm4f5\" (UID: \"e3f547af-2cf4-4821-ae28-e2087a814658\") " pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:55.787731 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.787580    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f547af-2cf4-4821-ae28-e2087a814658-tmp-dir\") pod \"node-resolver-qm4f5\" (UID: \"e3f547af-2cf4-4821-ae28-e2087a814658\") " pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:55.787731 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.787656    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft68x\" (UniqueName: \"kubernetes.io/projected/e3f547af-2cf4-4821-ae28-e2087a814658-kube-api-access-ft68x\") pod \"node-resolver-qm4f5\" (UID: \"e3f547af-2cf4-4821-ae28-e2087a814658\") " pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:55.888297 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.888254    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3f547af-2cf4-4821-ae28-e2087a814658-hosts-file\") pod \"node-resolver-qm4f5\" (UID: \"e3f547af-2cf4-4821-ae28-e2087a814658\") " pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:55.888461 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.888341    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f547af-2cf4-4821-ae28-e2087a814658-tmp-dir\") pod \"node-resolver-qm4f5\" (UID: \"e3f547af-2cf4-4821-ae28-e2087a814658\") " pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:55.888461 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.888373    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft68x\" (UniqueName: \"kubernetes.io/projected/e3f547af-2cf4-4821-ae28-e2087a814658-kube-api-access-ft68x\") pod \"node-resolver-qm4f5\" (UID: \"e3f547af-2cf4-4821-ae28-e2087a814658\") " pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:55.888773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.888749    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3f547af-2cf4-4821-ae28-e2087a814658-hosts-file\") pod \"node-resolver-qm4f5\" (UID: \"e3f547af-2cf4-4821-ae28-e2087a814658\") " pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:55.889046 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.889030    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f547af-2cf4-4821-ae28-e2087a814658-tmp-dir\") pod \"node-resolver-qm4f5\" (UID: \"e3f547af-2cf4-4821-ae28-e2087a814658\") " pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:55.897248 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:55.897206    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft68x\" (UniqueName: \"kubernetes.io/projected/e3f547af-2cf4-4821-ae28-e2087a814658-kube-api-access-ft68x\") pod \"node-resolver-qm4f5\" (UID: \"e3f547af-2cf4-4821-ae28-e2087a814658\") " pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:56.009733 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:56.009651    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qm4f5"
May 26 16:00:56.231860 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:56.231306    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:56.231860 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:56.231512    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:00:56.231860 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:56.231574    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:56.231860 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:56.231738    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:00:57.804975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:57.804933    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:57.805440 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:57.805095    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:57.805440 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:57.805155    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs podName:02034fc7-cc11-48cf-8eff-ae2384a2080e nodeName:}" failed. No retries permitted until 2026-05-26 16:01:05.805137001 +0000 UTC m=+17.186459021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs") pod "network-metrics-daemon-r88dc" (UID: "02034fc7-cc11-48cf-8eff-ae2384a2080e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:57.906231 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:57.905927    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:57.906231 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:57.906122    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:57.906231 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:57.906139    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:57.906231 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:57.906151    2570 projected.go:194] Error preparing data for projected volume kube-api-access-j6n5l for pod openshift-network-diagnostics/network-check-target-968nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:57.906231 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:57.906212    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l podName:49b37583-51bd-40e5-94c6-e1b191d0a891 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:05.906193287 +0000 UTC m=+17.287515292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6n5l" (UniqueName: "kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l") pod "network-check-target-968nq" (UID: "49b37583-51bd-40e5-94c6-e1b191d0a891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:58.231521 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:58.231422    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:00:58.231697 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:00:58.231425    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:00:58.231697 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:58.231559    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:00:58.231697 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:00:58.231644    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:00.231952 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:00.231710    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:00.232396 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:00.231718    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:00.232396 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:00.232033    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:00.232396 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:00.232125    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:02.231338 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:02.231297    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:02.231338 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:02.231325    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:02.231778 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:02.231474    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:02.231778 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:02.231562    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:04.231339 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:04.231311    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:04.231666 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:04.231316    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:04.231666 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:04.231408    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:04.231666 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:04.231479    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:05.858568 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:05.858529    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:05.859050 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:05.858732    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:05.859050 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:05.858815    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs podName:02034fc7-cc11-48cf-8eff-ae2384a2080e nodeName:}" failed. No retries permitted until 2026-05-26 16:01:21.858793387 +0000 UTC m=+33.240115392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs") pod "network-metrics-daemon-r88dc" (UID: "02034fc7-cc11-48cf-8eff-ae2384a2080e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:05.959916 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:05.959879    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:05.960088 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:05.960018    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:05.960088 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:05.960035    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:05.960088 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:05.960047    2570 projected.go:194] Error preparing data for projected volume kube-api-access-j6n5l for pod openshift-network-diagnostics/network-check-target-968nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:05.960218 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:05.960111    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l podName:49b37583-51bd-40e5-94c6-e1b191d0a891 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:21.96008997 +0000 UTC m=+33.341411993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6n5l" (UniqueName: "kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l") pod "network-check-target-968nq" (UID: "49b37583-51bd-40e5-94c6-e1b191d0a891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:06.231463 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:06.231383    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:06.231614 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:06.231382    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:06.231614 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:06.231507    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:06.231614 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:06.231593    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:08.231598 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:08.231558    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:08.232066 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:08.231558    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:08.232066 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:08.231700    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:08.232066 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:08.231760    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:08.961197 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:01:08.961153    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f547af_2cf4_4821_ae28_e2087a814658.slice/crio-f857d683c24d00cd3657cfb1d71ef28457a321a575f5489893283b83955a480a WatchSource:0}: Error finding container f857d683c24d00cd3657cfb1d71ef28457a321a575f5489893283b83955a480a: Status 404 returned error can't find the container with id f857d683c24d00cd3657cfb1d71ef28457a321a575f5489893283b83955a480a
May 26 16:01:09.390335 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:09.390127    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qm4f5" event={"ID":"e3f547af-2cf4-4821-ae28-e2087a814658","Type":"ContainerStarted","Data":"f857d683c24d00cd3657cfb1d71ef28457a321a575f5489893283b83955a480a"}
May 26 16:01:09.397106 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:09.397075    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-62bbh" event={"ID":"ddfb6a13-33b0-4ea4-ac38-ec92b912632e","Type":"ContainerStarted","Data":"98fef1322b25350e8b25073c9472078cb4f8e112516893d255c410b012fc49c6"}
May 26 16:01:09.402182 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:09.401979    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal" event={"ID":"84135cd3e968802e4af704bc56619b8f","Type":"ContainerStarted","Data":"40cb4a39852fea43369e13aae429f54c350a1c2d6c27ea23cfd674e3a380797b"}
May 26 16:01:09.404257 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:09.404194    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7bjv" event={"ID":"be3af2fc-3d53-46f3-b681-d665b14c11a8","Type":"ContainerStarted","Data":"d049b8f8f9ac4ca7ed82d95d923529ae0a66de29e3c4511afce99953c2c3e5af"}
May 26 16:01:09.418224 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:09.418164    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-62bbh" podStartSLOduration=2.026267653 podStartE2EDuration="20.418144574s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.569335712 +0000 UTC m=+1.950657718" lastFinishedPulling="2026-05-26 16:01:08.961212618 +0000 UTC m=+20.342534639" observedRunningTime="2026-05-26 16:01:09.417092848 +0000 UTC m=+20.798414874" watchObservedRunningTime="2026-05-26 16:01:09.418144574 +0000 UTC m=+20.799466600"
May 26 16:01:09.427924 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:09.427890    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" event={"ID":"270dc9ab-90fe-4504-8313-236f5a761696","Type":"ContainerStarted","Data":"946c2b09dc596229224a9f22af4c3ba52c860e50f738551f4e4237a00a658f43"}
May 26 16:01:09.449924 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:09.449855    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-k7bjv" podStartSLOduration=1.98342028 podStartE2EDuration="20.449834544s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.527156681 +0000 UTC m=+1.908478698" lastFinishedPulling="2026-05-26 16:01:08.993570954 +0000 UTC m=+20.374892962" observedRunningTime="2026-05-26 16:01:09.448132324 +0000 UTC m=+20.829454365" watchObservedRunningTime="2026-05-26 16:01:09.449834544 +0000 UTC m=+20.831156570"
May 26 16:01:09.450363 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:09.450306    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-89.ec2.internal" podStartSLOduration=20.450273609 podStartE2EDuration="20.450273609s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:01:09.43138313 +0000 UTC m=+20.812705155" watchObservedRunningTime="2026-05-26 16:01:09.450273609 +0000 UTC m=+20.831595635"
May 26 16:01:10.231583 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.231359    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:10.231739 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.231361    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:10.231739 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:10.231607    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:10.231739 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:10.231713    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:10.431295 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.431228    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mm79x" event={"ID":"bda7246b-9f6c-4339-86e9-2a97c2370213","Type":"ContainerStarted","Data":"08c32e71e0af2a2b9a64639b76b8882216bd4dcf6718eaf55b9caf3aea2bbcf1"}
May 26 16:01:10.432608 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.432581    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kbdlx" event={"ID":"9185f8d9-5823-4f14-bfb4-7d311127a7f1","Type":"ContainerStarted","Data":"801691e3c44e13bf603855d948d531caf507940f78cb27afa91d034dcf044544"}
May 26 16:01:10.435396 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.435369    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" event={"ID":"270dc9ab-90fe-4504-8313-236f5a761696","Type":"ContainerStarted","Data":"34dad0120daee220dd36efe1301860c67c9e738123ed1e605c47f7ec25023a7d"}
May 26 16:01:10.435508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.435404    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" event={"ID":"270dc9ab-90fe-4504-8313-236f5a761696","Type":"ContainerStarted","Data":"39775262af71e1f999db5a5c4aa344f7815a2fc451dcda798bd576f7b638946c"}
May 26 16:01:10.435508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.435420    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" event={"ID":"270dc9ab-90fe-4504-8313-236f5a761696","Type":"ContainerStarted","Data":"64ad6c4ee390345ff50da34ce2b92c87d3253588158480be936cd72cb920a5ef"}
May 26 16:01:10.435508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.435432    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" event={"ID":"270dc9ab-90fe-4504-8313-236f5a761696","Type":"ContainerStarted","Data":"795ee87849b82582d24d07225e8068921704aea32b3fed9b6e320e26a97fdbdb"}
May 26 16:01:10.435508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.435444    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" event={"ID":"270dc9ab-90fe-4504-8313-236f5a761696","Type":"ContainerStarted","Data":"25000e50649ca6ce582cd5ad9d2a576f76b26655c1215f80d760a2e048689169"}
May 26 16:01:10.436739 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.436717    2570 generic.go:358] "Generic (PLEG): container finished" podID="59beaf4f-19bd-4f7c-be73-07dad7f9fde2" containerID="a53b456f8e5409c1e781feff1acbba518940aa5e85b3c7326c204896be54b517" exitCode=0
May 26 16:01:10.436837 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.436770    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pjqf" event={"ID":"59beaf4f-19bd-4f7c-be73-07dad7f9fde2","Type":"ContainerDied","Data":"a53b456f8e5409c1e781feff1acbba518940aa5e85b3c7326c204896be54b517"}
May 26 16:01:10.438029 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.438008    2570 generic.go:358] "Generic (PLEG): container finished" podID="c9306cf6090e91ae784717834a4e427b" containerID="8fc175266397cb3becfaf8dcb5763f75c9979efece1bf55e5d2c3d3bbd8c5ed5" exitCode=0
May 26 16:01:10.438084 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.438070    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal" event={"ID":"c9306cf6090e91ae784717834a4e427b","Type":"ContainerDied","Data":"8fc175266397cb3becfaf8dcb5763f75c9979efece1bf55e5d2c3d3bbd8c5ed5"}
May 26 16:01:10.439393 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.439306    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qm4f5" event={"ID":"e3f547af-2cf4-4821-ae28-e2087a814658","Type":"ContainerStarted","Data":"fedecbfc0b6e25403aeeff35baeb28f02f8f6a5f6a2e9c2da2cdca63def0464c"}
May 26 16:01:10.441261 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.441211    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh" event={"ID":"7e844313-be2a-4801-966e-6a1257eaeef4","Type":"ContainerStarted","Data":"788a996dd26c0781cdba7444c782e5b8c9ca9de03d927deaef5f142a959ebb32"}
May 26 16:01:10.443556 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.443509    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mm79x" podStartSLOduration=3.046841223 podStartE2EDuration="21.443494818s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.508132901 +0000 UTC m=+1.889454904" lastFinishedPulling="2026-05-26 16:01:08.904786481 +0000 UTC m=+20.286108499" observedRunningTime="2026-05-26 16:01:10.443155552 +0000 UTC m=+21.824477577" watchObservedRunningTime="2026-05-26 16:01:10.443494818 +0000 UTC m=+21.824816844"
May 26 16:01:10.457271 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.457217    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-qm4f5" podStartSLOduration=15.457200392 podStartE2EDuration="15.457200392s" podCreationTimestamp="2026-05-26 16:00:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:01:10.456959073 +0000 UTC m=+21.838281124" watchObservedRunningTime="2026-05-26 16:01:10.457200392 +0000 UTC m=+21.838522419"
May 26 16:01:10.499270 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.499158    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kbdlx" podStartSLOduration=3.074951302 podStartE2EDuration="21.499141907s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.480597603 +0000 UTC m=+1.861919606" lastFinishedPulling="2026-05-26 16:01:08.904788193 +0000 UTC m=+20.286110211" observedRunningTime="2026-05-26 16:01:10.482224689 +0000 UTC m=+21.863546717" watchObservedRunningTime="2026-05-26 16:01:10.499141907 +0000 UTC m=+21.880463933"
May 26 16:01:10.520077 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:10.520050    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 16:01:11.206453 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:11.206347    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T16:01:10.520074494Z","UUID":"d7e39125-a608-47f9-9172-693d8d6fae5f","Handler":null,"Name":"","Endpoint":""}
May 26 16:01:11.208726 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:11.208694    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 16:01:11.208726 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:11.208725    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 16:01:11.445025 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:11.444986    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal" event={"ID":"c9306cf6090e91ae784717834a4e427b","Type":"ContainerStarted","Data":"d6b3f610ab71c1c7886e05d232039d55f2502ca9abe8e5b3b253919d4ae119ec"}
May 26 16:01:11.446745 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:11.446721    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh" event={"ID":"7e844313-be2a-4801-966e-6a1257eaeef4","Type":"ContainerStarted","Data":"b01236e7ffaf1ae74b7c47a1449f33c3a95d839ccd0bb8d639c0a91d8bb3180f"}
May 26 16:01:11.448079 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:11.447986    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-27bnz" event={"ID":"557f56c9-c491-4f35-99de-92a9cdc50f23","Type":"ContainerStarted","Data":"f8b666e1b745ea1fcbe62d78dfb3fc51aa62c509066c4f11f80516736d406510"}
May 26 16:01:11.470930 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:11.470894    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-89.ec2.internal" podStartSLOduration=22.470879773 podStartE2EDuration="22.470879773s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:01:11.470795487 +0000 UTC m=+22.852117514" watchObservedRunningTime="2026-05-26 16:01:11.470879773 +0000 UTC m=+22.852201801"
May 26 16:01:11.484702 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:11.484654    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-27bnz" podStartSLOduration=4.064661651 podStartE2EDuration="22.484636519s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.541195016 +0000 UTC m=+1.922517022" lastFinishedPulling="2026-05-26 16:01:08.961169873 +0000 UTC m=+20.342491890" observedRunningTime="2026-05-26 16:01:11.48448229 +0000 UTC m=+22.865804315" watchObservedRunningTime="2026-05-26 16:01:11.484636519 +0000 UTC m=+22.865958542"
May 26 16:01:12.232084 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:12.232002    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:12.232227 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:12.232013    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:12.232227 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:12.232111    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:12.232227 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:12.232189    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:12.453714 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:12.453669    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" event={"ID":"270dc9ab-90fe-4504-8313-236f5a761696","Type":"ContainerStarted","Data":"7bebce75d25ac471be6d42ca331b624814a6255c5bf2cbed2f7edb38e84020c9"}
May 26 16:01:12.455760 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:12.455725    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh" event={"ID":"7e844313-be2a-4801-966e-6a1257eaeef4","Type":"ContainerStarted","Data":"5d6ff091158a2235f624ff9363666f81a736061d583393872ad5b1cf78bb166d"}
May 26 16:01:12.473050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:12.473006    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cnkh" podStartSLOduration=2.568726186 podStartE2EDuration="23.472980425s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.555440177 +0000 UTC m=+1.936762184" lastFinishedPulling="2026-05-26 16:01:11.459694406 +0000 UTC m=+22.841016423" observedRunningTime="2026-05-26 16:01:12.472809798 +0000 UTC m=+23.854131822" watchObservedRunningTime="2026-05-26 16:01:12.472980425 +0000 UTC m=+23.854302450"
May 26 16:01:14.231267 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:14.231231    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:14.231675 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:14.231232    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:14.231675 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:14.231346    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:14.231675 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:14.231458    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:14.461877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:14.461852    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" event={"ID":"270dc9ab-90fe-4504-8313-236f5a761696","Type":"ContainerStarted","Data":"5228a0a17a9d9496624be35c48749291b2939d32ad5042a3cbcae440a218c582"}
May 26 16:01:14.467342 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:14.467323    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:01:14.485483 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:14.485316    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:01:14.494842 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:14.494701    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s" podStartSLOduration=6.660518214 podStartE2EDuration="25.494685367s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.466831789 +0000 UTC m=+1.848153797" lastFinishedPulling="2026-05-26 16:01:09.300998944 +0000 UTC m=+20.682320950" observedRunningTime="2026-05-26 16:01:14.493431277 +0000 UTC m=+25.874753302" watchObservedRunningTime="2026-05-26 16:01:14.494685367 +0000 UTC m=+25.876007393"
May 26 16:01:14.737012 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:14.736978    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:01:14.737620 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:14.737595    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:01:15.465060 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:15.465026    2570 generic.go:358] "Generic (PLEG): container finished" podID="59beaf4f-19bd-4f7c-be73-07dad7f9fde2" containerID="83a990f8dbe031857f0e2b9bfc718398c57a510ce10f2a7ee523a51b512bf3c0" exitCode=0
May 26 16:01:15.466176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:15.465095    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pjqf" event={"ID":"59beaf4f-19bd-4f7c-be73-07dad7f9fde2","Type":"ContainerDied","Data":"83a990f8dbe031857f0e2b9bfc718398c57a510ce10f2a7ee523a51b512bf3c0"}
May 26 16:01:15.466176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:15.465311    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:01:15.466176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:15.465386    2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 16:01:15.466176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:15.465974    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:01:15.467134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:15.466202    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kbdlx"
May 26 16:01:15.483625 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:15.483595    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:01:16.221197 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:16.221161    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r88dc"]
May 26 16:01:16.221383 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:16.221309    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:16.221426 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:16.221396    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:16.223784 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:16.223754    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-968nq"]
May 26 16:01:16.223895 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:16.223833    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:16.223938 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:16.223896    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:16.363169 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:16.363102    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:01:17.469752 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:17.469715    2570 generic.go:358] "Generic (PLEG): container finished" podID="59beaf4f-19bd-4f7c-be73-07dad7f9fde2" containerID="356ae3f79b85b1aa5fc35f1a3460fbc031a2a29d2493876db15cc1b047a4ae38" exitCode=0
May 26 16:01:17.470255 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:17.469796    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pjqf" event={"ID":"59beaf4f-19bd-4f7c-be73-07dad7f9fde2","Type":"ContainerDied","Data":"356ae3f79b85b1aa5fc35f1a3460fbc031a2a29d2493876db15cc1b047a4ae38"}
May 26 16:01:18.231818 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:18.231596    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:18.231932 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:18.231601    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:18.231975 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:18.231923    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:18.232041 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:18.232022    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:18.473708 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:18.473675    2570 generic.go:358] "Generic (PLEG): container finished" podID="59beaf4f-19bd-4f7c-be73-07dad7f9fde2" containerID="c2447728d14c691d06daacd34d342db0556fff462ea0abf96ba1d086279455e4" exitCode=0
May 26 16:01:18.474071 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:18.473733    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pjqf" event={"ID":"59beaf4f-19bd-4f7c-be73-07dad7f9fde2","Type":"ContainerDied","Data":"c2447728d14c691d06daacd34d342db0556fff462ea0abf96ba1d086279455e4"}
May 26 16:01:20.231220 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:20.231180    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:20.231668 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:20.231320    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-968nq" podUID="49b37583-51bd-40e5-94c6-e1b191d0a891"
May 26 16:01:20.231668 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:20.231180    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:20.231794 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:20.231681    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:01:21.876985 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:21.876951    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:21.877586 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:21.877071    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:21.877586 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:21.877132    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs podName:02034fc7-cc11-48cf-8eff-ae2384a2080e nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.877111552 +0000 UTC m=+65.258433558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs") pod "network-metrics-daemon-r88dc" (UID: "02034fc7-cc11-48cf-8eff-ae2384a2080e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:21.963212 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:21.963144    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-89.ec2.internal" event="NodeReady"
May 26 16:01:21.963371 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:21.963265    2570 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 16:01:21.977637 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:21.977610    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:21.977740 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:21.977715    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:21.977740 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:21.977728    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:21.977740 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:21.977738    2570 projected.go:194] Error preparing data for projected volume kube-api-access-j6n5l for pod openshift-network-diagnostics/network-check-target-968nq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:21.977853 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:21.977779    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l podName:49b37583-51bd-40e5-94c6-e1b191d0a891 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.977767303 +0000 UTC m=+65.359089307 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6n5l" (UniqueName: "kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l") pod "network-check-target-968nq" (UID: "49b37583-51bd-40e5-94c6-e1b191d0a891") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:22.005762 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.005736    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2lbjs"]
May 26 16:01:22.017563 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.017542    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xd8kr"]
May 26 16:01:22.017741 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.017718    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.019990 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.019969    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 16:01:22.020217 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.020197    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 16:01:22.020333 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.020255    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lppkw\""
May 26 16:01:22.033795 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.033777    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2lbjs"]
May 26 16:01:22.033908 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.033800    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xd8kr"]
May 26 16:01:22.033969 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.033929    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:22.036028 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.036008    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 16:01:22.036109 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.036030    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-22s8h\""
May 26 16:01:22.036109 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.036084    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 16:01:22.036313 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.036298    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 16:01:22.179442 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.179412    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea83ed2-100d-4de2-a94d-5aed66a964a7-tmp-dir\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.179442 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.179461    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g55gr\" (UniqueName: \"kubernetes.io/projected/9ea83ed2-100d-4de2-a94d-5aed66a964a7-kube-api-access-g55gr\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.179700 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.179483    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5lq\" (UniqueName: \"kubernetes.io/projected/a560b45b-1006-463e-8922-5b41684d6458-kube-api-access-6c5lq\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:22.179700 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.179553    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea83ed2-100d-4de2-a94d-5aed66a964a7-config-volume\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.179700 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.179628    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:22.179700 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.179685    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.231938 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.231849    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:22.231938 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.231879    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:22.235027 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.234887    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:01:22.235027 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.234900    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:01:22.235027 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.234903    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:01:22.235027 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.234966    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8ddfl\""
May 26 16:01:22.235332 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.235208    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45wxn\""
May 26 16:01:22.280379 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.280353    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea83ed2-100d-4de2-a94d-5aed66a964a7-tmp-dir\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.280532 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.280411    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g55gr\" (UniqueName: \"kubernetes.io/projected/9ea83ed2-100d-4de2-a94d-5aed66a964a7-kube-api-access-g55gr\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.280532 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.280441    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5lq\" (UniqueName: \"kubernetes.io/projected/a560b45b-1006-463e-8922-5b41684d6458-kube-api-access-6c5lq\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:22.280532 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.280473    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea83ed2-100d-4de2-a94d-5aed66a964a7-config-volume\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.280532 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.280500    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:22.280729 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.280542    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.280729 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:22.280685    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:22.280829 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:22.280745    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls podName:9ea83ed2-100d-4de2-a94d-5aed66a964a7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:22.780723526 +0000 UTC m=+34.162045547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls") pod "dns-default-2lbjs" (UID: "9ea83ed2-100d-4de2-a94d-5aed66a964a7") : secret "dns-default-metrics-tls" not found
May 26 16:01:22.281004 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:22.280985    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:22.281086 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.281031    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea83ed2-100d-4de2-a94d-5aed66a964a7-tmp-dir\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.281086 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:22.281051    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert podName:a560b45b-1006-463e-8922-5b41684d6458 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:22.781032702 +0000 UTC m=+34.162354719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert") pod "ingress-canary-xd8kr" (UID: "a560b45b-1006-463e-8922-5b41684d6458") : secret "canary-serving-cert" not found
May 26 16:01:22.281520 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.281474    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea83ed2-100d-4de2-a94d-5aed66a964a7-config-volume\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.293192 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.293168    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g55gr\" (UniqueName: \"kubernetes.io/projected/9ea83ed2-100d-4de2-a94d-5aed66a964a7-kube-api-access-g55gr\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.293376 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.293350    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5lq\" (UniqueName: \"kubernetes.io/projected/a560b45b-1006-463e-8922-5b41684d6458-kube-api-access-6c5lq\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:22.783648 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.783613    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:22.783840 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:22.783665    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:22.783840 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:22.783780    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:22.783840 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:22.783780    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:22.783982 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:22.783848    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls podName:9ea83ed2-100d-4de2-a94d-5aed66a964a7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:23.783833039 +0000 UTC m=+35.165155041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls") pod "dns-default-2lbjs" (UID: "9ea83ed2-100d-4de2-a94d-5aed66a964a7") : secret "dns-default-metrics-tls" not found
May 26 16:01:22.783982 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:22.783863    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert podName:a560b45b-1006-463e-8922-5b41684d6458 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:23.78385773 +0000 UTC m=+35.165179733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert") pod "ingress-canary-xd8kr" (UID: "a560b45b-1006-463e-8922-5b41684d6458") : secret "canary-serving-cert" not found
May 26 16:01:23.618533 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.618496    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jqhkk"]
May 26 16:01:23.651243 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.651206    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jqhkk"]
May 26 16:01:23.651423 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.651405    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.653974 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.653950    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 16:01:23.792219 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.792132    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/799f1c88-c483-4ec4-b68c-4b9e2850bcc8-kubelet-config\") pod \"global-pull-secret-syncer-jqhkk\" (UID: \"799f1c88-c483-4ec4-b68c-4b9e2850bcc8\") " pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.792219 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.792198    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:23.792474 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.792231    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/799f1c88-c483-4ec4-b68c-4b9e2850bcc8-dbus\") pod \"global-pull-secret-syncer-jqhkk\" (UID: \"799f1c88-c483-4ec4-b68c-4b9e2850bcc8\") " pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.792474 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.792272    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:23.792474 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:23.792341    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:23.792474 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.792342    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/799f1c88-c483-4ec4-b68c-4b9e2850bcc8-original-pull-secret\") pod \"global-pull-secret-syncer-jqhkk\" (UID: \"799f1c88-c483-4ec4-b68c-4b9e2850bcc8\") " pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.792474 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:23.792401    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert podName:a560b45b-1006-463e-8922-5b41684d6458 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.792382166 +0000 UTC m=+37.173704178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert") pod "ingress-canary-xd8kr" (UID: "a560b45b-1006-463e-8922-5b41684d6458") : secret "canary-serving-cert" not found
May 26 16:01:23.792474 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:23.792429    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:23.792745 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:23.792485    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls podName:9ea83ed2-100d-4de2-a94d-5aed66a964a7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:25.792467919 +0000 UTC m=+37.173789950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls") pod "dns-default-2lbjs" (UID: "9ea83ed2-100d-4de2-a94d-5aed66a964a7") : secret "dns-default-metrics-tls" not found
May 26 16:01:23.893413 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.893379    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/799f1c88-c483-4ec4-b68c-4b9e2850bcc8-kubelet-config\") pod \"global-pull-secret-syncer-jqhkk\" (UID: \"799f1c88-c483-4ec4-b68c-4b9e2850bcc8\") " pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.893575 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.893442    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/799f1c88-c483-4ec4-b68c-4b9e2850bcc8-dbus\") pod \"global-pull-secret-syncer-jqhkk\" (UID: \"799f1c88-c483-4ec4-b68c-4b9e2850bcc8\") " pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.893575 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.893515    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/799f1c88-c483-4ec4-b68c-4b9e2850bcc8-kubelet-config\") pod \"global-pull-secret-syncer-jqhkk\" (UID: \"799f1c88-c483-4ec4-b68c-4b9e2850bcc8\") " pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.893575 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.893525    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/799f1c88-c483-4ec4-b68c-4b9e2850bcc8-original-pull-secret\") pod \"global-pull-secret-syncer-jqhkk\" (UID: \"799f1c88-c483-4ec4-b68c-4b9e2850bcc8\") " pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.893719 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.893696    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/799f1c88-c483-4ec4-b68c-4b9e2850bcc8-dbus\") pod \"global-pull-secret-syncer-jqhkk\" (UID: \"799f1c88-c483-4ec4-b68c-4b9e2850bcc8\") " pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.896060 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.896040    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/799f1c88-c483-4ec4-b68c-4b9e2850bcc8-original-pull-secret\") pod \"global-pull-secret-syncer-jqhkk\" (UID: \"799f1c88-c483-4ec4-b68c-4b9e2850bcc8\") " pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:23.963119 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:23.963091    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqhkk"
May 26 16:01:24.332873 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:24.332649    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jqhkk"]
May 26 16:01:24.392103 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:01:24.392068    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799f1c88_c483_4ec4_b68c_4b9e2850bcc8.slice/crio-8bcaedbb6a6b060f340a0d5f26c8a26bae3dedaee3b57e187df0987fa2ca619e WatchSource:0}: Error finding container 8bcaedbb6a6b060f340a0d5f26c8a26bae3dedaee3b57e187df0987fa2ca619e: Status 404 returned error can't find the container with id 8bcaedbb6a6b060f340a0d5f26c8a26bae3dedaee3b57e187df0987fa2ca619e
May 26 16:01:24.486578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:24.486547    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jqhkk" event={"ID":"799f1c88-c483-4ec4-b68c-4b9e2850bcc8","Type":"ContainerStarted","Data":"8bcaedbb6a6b060f340a0d5f26c8a26bae3dedaee3b57e187df0987fa2ca619e"}
May 26 16:01:25.491443 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:25.491399    2570 generic.go:358] "Generic (PLEG): container finished" podID="59beaf4f-19bd-4f7c-be73-07dad7f9fde2" containerID="6d457bd4f03d11e4eecfb78a499e41d7074dbcb1b0a9f31290920bb9273b2912" exitCode=0
May 26 16:01:25.492005 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:25.491461    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pjqf" event={"ID":"59beaf4f-19bd-4f7c-be73-07dad7f9fde2","Type":"ContainerDied","Data":"6d457bd4f03d11e4eecfb78a499e41d7074dbcb1b0a9f31290920bb9273b2912"}
May 26 16:01:25.809469 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:25.809436    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:25.809645 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:25.809533    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:25.809645 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:25.809595    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:25.809759 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:25.809650    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:25.809759 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:25.809677    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls podName:9ea83ed2-100d-4de2-a94d-5aed66a964a7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.809655946 +0000 UTC m=+41.190977951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls") pod "dns-default-2lbjs" (UID: "9ea83ed2-100d-4de2-a94d-5aed66a964a7") : secret "dns-default-metrics-tls" not found
May 26 16:01:25.809759 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:25.809705    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert podName:a560b45b-1006-463e-8922-5b41684d6458 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.809687108 +0000 UTC m=+41.191009134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert") pod "ingress-canary-xd8kr" (UID: "a560b45b-1006-463e-8922-5b41684d6458") : secret "canary-serving-cert" not found
May 26 16:01:26.496724 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:26.496686    2570 generic.go:358] "Generic (PLEG): container finished" podID="59beaf4f-19bd-4f7c-be73-07dad7f9fde2" containerID="2f00c49cb99cc3b3b1c5eaf114afbbe51bf056574daf1a74d86660edb1d51a12" exitCode=0
May 26 16:01:26.497117 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:26.496754    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pjqf" event={"ID":"59beaf4f-19bd-4f7c-be73-07dad7f9fde2","Type":"ContainerDied","Data":"2f00c49cb99cc3b3b1c5eaf114afbbe51bf056574daf1a74d86660edb1d51a12"}
May 26 16:01:27.502868 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:27.502668    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pjqf" event={"ID":"59beaf4f-19bd-4f7c-be73-07dad7f9fde2","Type":"ContainerStarted","Data":"ebf28543d8c1e30af75da7f9e6164aba347a6252297ce74d1274063d9209f01b"}
May 26 16:01:27.523917 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:27.523857    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2pjqf" podStartSLOduration=4.552075981 podStartE2EDuration="38.523841907s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:00:50.446758924 +0000 UTC m=+1.828080931" lastFinishedPulling="2026-05-26 16:01:24.418524854 +0000 UTC m=+35.799846857" observedRunningTime="2026-05-26 16:01:27.523833348 +0000 UTC m=+38.905155376" watchObservedRunningTime="2026-05-26 16:01:27.523841907 +0000 UTC m=+38.905163931"
May 26 16:01:29.506779 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:29.506745    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jqhkk" event={"ID":"799f1c88-c483-4ec4-b68c-4b9e2850bcc8","Type":"ContainerStarted","Data":"ed38977b6ec3a4d533d6b3f2e1ebfe6a479ad69d7abaccde21201393e3b91a93"}
May 26 16:01:29.520893 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:29.520844    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jqhkk" podStartSLOduration=2.350917459 podStartE2EDuration="6.520828718s" podCreationTimestamp="2026-05-26 16:01:23 +0000 UTC" firstStartedPulling="2026-05-26 16:01:24.397005151 +0000 UTC m=+35.778327157" lastFinishedPulling="2026-05-26 16:01:28.566916413 +0000 UTC m=+39.948238416" observedRunningTime="2026-05-26 16:01:29.520541933 +0000 UTC m=+40.901863959" watchObservedRunningTime="2026-05-26 16:01:29.520828718 +0000 UTC m=+40.902150745"
May 26 16:01:29.840824 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:29.840737    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:29.840824 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:29.840784    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:29.841016 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:29.840886    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:29.841016 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:29.840891    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:29.841016 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:29.840934    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls podName:9ea83ed2-100d-4de2-a94d-5aed66a964a7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:37.840921169 +0000 UTC m=+49.222243173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls") pod "dns-default-2lbjs" (UID: "9ea83ed2-100d-4de2-a94d-5aed66a964a7") : secret "dns-default-metrics-tls" not found
May 26 16:01:29.841016 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:29.840947    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert podName:a560b45b-1006-463e-8922-5b41684d6458 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:37.84094106 +0000 UTC m=+49.222263063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert") pod "ingress-canary-xd8kr" (UID: "a560b45b-1006-463e-8922-5b41684d6458") : secret "canary-serving-cert" not found
May 26 16:01:37.896494 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:37.896455    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:37.897057 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:37.896525    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:37.897057 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:37.896609    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:37.897057 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:37.896621    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:37.897057 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:37.896658    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert podName:a560b45b-1006-463e-8922-5b41684d6458 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.896645469 +0000 UTC m=+65.277967473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert") pod "ingress-canary-xd8kr" (UID: "a560b45b-1006-463e-8922-5b41684d6458") : secret "canary-serving-cert" not found
May 26 16:01:37.897057 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:37.896688    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls podName:9ea83ed2-100d-4de2-a94d-5aed66a964a7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:53.896670899 +0000 UTC m=+65.277992919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls") pod "dns-default-2lbjs" (UID: "9ea83ed2-100d-4de2-a94d-5aed66a964a7") : secret "dns-default-metrics-tls" not found
May 26 16:01:47.480114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:47.480087    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8fv6s"
May 26 16:01:53.902576 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:53.902539    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:01:53.902576 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:53.902575    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:01:53.903021 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:53.902615    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:01:53.903021 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:53.902685    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:53.903021 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:53.902690    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:53.903021 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:53.902735    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert podName:a560b45b-1006-463e-8922-5b41684d6458 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:25.902718779 +0000 UTC m=+97.284040785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert") pod "ingress-canary-xd8kr" (UID: "a560b45b-1006-463e-8922-5b41684d6458") : secret "canary-serving-cert" not found
May 26 16:01:53.903021 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:53.902749    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls podName:9ea83ed2-100d-4de2-a94d-5aed66a964a7 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:25.902743566 +0000 UTC m=+97.284065568 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls") pod "dns-default-2lbjs" (UID: "9ea83ed2-100d-4de2-a94d-5aed66a964a7") : secret "dns-default-metrics-tls" not found
May 26 16:01:53.904604 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:53.904585    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:01:53.913385 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:53.913369    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 16:01:53.913426 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:01:53.913409    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs podName:02034fc7-cc11-48cf-8eff-ae2384a2080e nodeName:}" failed. No retries permitted until 2026-05-26 16:02:57.913397893 +0000 UTC m=+129.294719895 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs") pod "network-metrics-daemon-r88dc" (UID: "02034fc7-cc11-48cf-8eff-ae2384a2080e") : secret "metrics-daemon-secret" not found
May 26 16:01:54.003589 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:54.003545    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:54.005548 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:54.005533    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:01:54.016358 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:54.016342    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:01:54.029036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:54.029015    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6n5l\" (UniqueName: \"kubernetes.io/projected/49b37583-51bd-40e5-94c6-e1b191d0a891-kube-api-access-j6n5l\") pod \"network-check-target-968nq\" (UID: \"49b37583-51bd-40e5-94c6-e1b191d0a891\") " pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:54.046059 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:54.046039    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45wxn\""
May 26 16:01:54.054002 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:54.053985    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:54.185241 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:54.185167    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-968nq"]
May 26 16:01:54.188602 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:01:54.188572    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b37583_51bd_40e5_94c6_e1b191d0a891.slice/crio-d23aa17148e20461712efd007f17bfdd6d3e7dc718634075601f205ab5ec87d0 WatchSource:0}: Error finding container d23aa17148e20461712efd007f17bfdd6d3e7dc718634075601f205ab5ec87d0: Status 404 returned error can't find the container with id d23aa17148e20461712efd007f17bfdd6d3e7dc718634075601f205ab5ec87d0
May 26 16:01:54.552987 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:54.552906    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-968nq" event={"ID":"49b37583-51bd-40e5-94c6-e1b191d0a891","Type":"ContainerStarted","Data":"d23aa17148e20461712efd007f17bfdd6d3e7dc718634075601f205ab5ec87d0"}
May 26 16:01:58.561026 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:58.560990    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-968nq" event={"ID":"49b37583-51bd-40e5-94c6-e1b191d0a891","Type":"ContainerStarted","Data":"64720f4b3ea7587e48192e4a6ad0997d6a7e7b61b85cb24f18ba63069b6d9b7a"}
May 26 16:01:58.561496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:58.561111    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:01:58.577394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:01:58.577350    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-968nq" podStartSLOduration=65.998471615 podStartE2EDuration="1m9.577339312s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:01:54.190303076 +0000 UTC m=+65.571625083" lastFinishedPulling="2026-05-26 16:01:57.769170764 +0000 UTC m=+69.150492780" observedRunningTime="2026-05-26 16:01:58.576731699 +0000 UTC m=+69.958053725" watchObservedRunningTime="2026-05-26 16:01:58.577339312 +0000 UTC m=+69.958661331"
May 26 16:02:25.926604 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:02:25.926552    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:02:25.927027 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:02:25.926637    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:02:25.927027 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:02:25.926709    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:25.927027 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:02:25.926778    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls podName:9ea83ed2-100d-4de2-a94d-5aed66a964a7 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:29.926760366 +0000 UTC m=+161.308082369 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls") pod "dns-default-2lbjs" (UID: "9ea83ed2-100d-4de2-a94d-5aed66a964a7") : secret "dns-default-metrics-tls" not found
May 26 16:02:25.927027 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:02:25.926712    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:25.927027 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:02:25.926855    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert podName:a560b45b-1006-463e-8922-5b41684d6458 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:29.926841499 +0000 UTC m=+161.308163502 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert") pod "ingress-canary-xd8kr" (UID: "a560b45b-1006-463e-8922-5b41684d6458") : secret "canary-serving-cert" not found
May 26 16:02:29.565611 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:02:29.565578    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-968nq"
May 26 16:02:57.944289 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:02:57.944237    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:02:57.944893 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:02:57.944407    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 16:02:57.944893 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:02:57.944505    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs podName:02034fc7-cc11-48cf-8eff-ae2384a2080e nodeName:}" failed. No retries permitted until 2026-05-26 16:04:59.944483464 +0000 UTC m=+251.325805480 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs") pod "network-metrics-daemon-r88dc" (UID: "02034fc7-cc11-48cf-8eff-ae2384a2080e") : secret "metrics-daemon-secret" not found
May 26 16:03:25.027672 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:25.027627    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2lbjs" podUID="9ea83ed2-100d-4de2-a94d-5aed66a964a7"
May 26 16:03:25.041826 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:25.041778    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-xd8kr" podUID="a560b45b-1006-463e-8922-5b41684d6458"
May 26 16:03:25.250638 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:25.250592    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-r88dc" podUID="02034fc7-cc11-48cf-8eff-ae2384a2080e"
May 26 16:03:25.720915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:25.720888    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2lbjs"
May 26 16:03:29.954095 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:29.954060    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:03:29.954095 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:29.954096    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:03:29.954597 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:29.954198    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:03:29.954597 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:29.954218    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:03:29.954597 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:29.954249    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls podName:9ea83ed2-100d-4de2-a94d-5aed66a964a7 nodeName:}" failed. No retries permitted until 2026-05-26 16:05:31.954236337 +0000 UTC m=+283.335558341 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls") pod "dns-default-2lbjs" (UID: "9ea83ed2-100d-4de2-a94d-5aed66a964a7") : secret "dns-default-metrics-tls" not found
May 26 16:03:29.954597 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:29.954312    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert podName:a560b45b-1006-463e-8922-5b41684d6458 nodeName:}" failed. No retries permitted until 2026-05-26 16:05:31.954272689 +0000 UTC m=+283.335594706 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert") pod "ingress-canary-xd8kr" (UID: "a560b45b-1006-463e-8922-5b41684d6458") : secret "canary-serving-cert" not found
May 26 16:03:31.724715 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.724684    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-5nqbd"]
May 26 16:03:31.727471 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.727455    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.729569 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.729546    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 26 16:03:31.729749 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.729730    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 26 16:03:31.730121 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.730104    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 16:03:31.730724 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.730705    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 16:03:31.731619 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.731606    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6jzjv\""
May 26 16:03:31.734496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.734479    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 26 16:03:31.735959 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.735940    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-5nqbd"]
May 26 16:03:31.828327 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.828296    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"]
May 26 16:03:31.831121 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.831105    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f76bdfcbb-7twj9"]
May 26 16:03:31.831291 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.831259    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:31.833462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.833446    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 26 16:03:31.833558 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.833474    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v6kfn\""
May 26 16:03:31.833558 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.833528    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 26 16:03:31.833558 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.833534    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 26 16:03:31.834507 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.834493    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:31.836552 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.836535    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 26 16:03:31.836683 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.836548    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 26 16:03:31.836683 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.836675    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 26 16:03:31.836836 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.836606    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 26 16:03:31.836836 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.836618    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 26 16:03:31.836836 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.836638    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 26 16:03:31.836836 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.836554    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qn9n8\""
May 26 16:03:31.842095 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.842076    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"]
May 26 16:03:31.854231 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.854209    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f76bdfcbb-7twj9"]
May 26 16:03:31.866379 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.866361    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-snapshots\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.866462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.866398    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.866462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.866416    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-serving-cert\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.866462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.866434    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-service-ca-bundle\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.866573 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.866507    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-tmp\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.866573 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.866568    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qql4z\" (UniqueName: \"kubernetes.io/projected/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-kube-api-access-qql4z\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.919181 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.919157    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm"]
May 26 16:03:31.921912 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.921895    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm"
May 26 16:03:31.922021 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.922000    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"]
May 26 16:03:31.924161 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.924145    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kdzmx\""
May 26 16:03:31.924822 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.924807    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:31.926791 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.926770    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 26 16:03:31.926907 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.926792    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 26 16:03:31.926907 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.926790    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vn7fq\""
May 26 16:03:31.926907 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.926777    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 26 16:03:31.927160 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.927138    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 26 16:03:31.931437 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.931419    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm"]
May 26 16:03:31.935080 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.935059    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"]
May 26 16:03:31.967420 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967393    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:31.967420 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967423    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-stats-auth\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:31.967581 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967441    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk6hx\" (UniqueName: \"kubernetes.io/projected/b0ad9be8-9903-47e5-bc99-2c61845cce5e-kube-api-access-lk6hx\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:31.967581 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967460    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:31.967581 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967501    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qql4z\" (UniqueName: \"kubernetes.io/projected/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-kube-api-access-qql4z\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.967581 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967545    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-snapshots\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.967581 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967574    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.967790 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967589    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-serving-cert\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.967790 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967611    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-service-ca-bundle\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.967790 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967633    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-tmp\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.967790 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:31.967790 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.967713    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-default-certificate\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:31.968113 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.968080    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2vr\" (UniqueName: \"kubernetes.io/projected/a218303e-8766-4357-9cfe-e957492ea3b0-kube-api-access-hj2vr\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:31.968382 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.968359    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-tmp\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.968569 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.968545    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-service-ca-bundle\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.968650 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.968634    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-snapshots\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.972544 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.972435    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.972790 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.972774    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-serving-cert\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:31.981662 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:31.981617    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qql4z\" (UniqueName: \"kubernetes.io/projected/6fe2ee2b-83d4-401a-a6e1-389caeeb7e09-kube-api-access-qql4z\") pod \"insights-operator-78bdfb888c-5nqbd\" (UID: \"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09\") " pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:32.036727 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.036701    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-5nqbd"
May 26 16:03:32.069475 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069445    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.069637 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069481    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-stats-auth\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.069637 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069507    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lk6hx\" (UniqueName: \"kubernetes.io/projected/b0ad9be8-9903-47e5-bc99-2c61845cce5e-kube-api-access-lk6hx\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.069746 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069635    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.069746 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.069677    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:32.569652972 +0000 UTC m=+163.950974994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : configmap references non-existent config key: service-ca.crt
May 26 16:03:32.069746 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.069716    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:03:32.069746 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069716    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa82f87f-2f4a-4cd6-98cd-c72f77130748-config\") pod \"service-ca-operator-57b5b68b8-nvhtb\" (UID: \"aa82f87f-2f4a-4cd6-98cd-c72f77130748\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.069967 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.069765    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:32.569753473 +0000 UTC m=+163.951075477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : secret "router-metrics-certs-default" not found
May 26 16:03:32.069967 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069815    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6kvh\" (UniqueName: \"kubernetes.io/projected/aa82f87f-2f4a-4cd6-98cd-c72f77130748-kube-api-access-c6kvh\") pod \"service-ca-operator-57b5b68b8-nvhtb\" (UID: \"aa82f87f-2f4a-4cd6-98cd-c72f77130748\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.069967 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069877    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:32.069967 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069901    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-default-certificate\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.069967 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069933    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa82f87f-2f4a-4cd6-98cd-c72f77130748-serving-cert\") pod \"service-ca-operator-57b5b68b8-nvhtb\" (UID: \"aa82f87f-2f4a-4cd6-98cd-c72f77130748\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.070210 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.069969    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkxg\" (UniqueName: \"kubernetes.io/projected/ce52bd45-0e01-4b65-814b-294a61999d19-kube-api-access-gvkxg\") pod \"network-check-source-7f44b6dd87-xgtzm\" (UID: \"ce52bd45-0e01-4b65-814b-294a61999d19\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm"
May 26 16:03:32.070210 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.069987    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:03:32.070210 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.070003    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2vr\" (UniqueName: \"kubernetes.io/projected/a218303e-8766-4357-9cfe-e957492ea3b0-kube-api-access-hj2vr\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:32.070210 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.070073    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls podName:a218303e-8766-4357-9cfe-e957492ea3b0 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:32.570055657 +0000 UTC m=+163.951377674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls") pod "cluster-samples-operator-697587599-9cs29" (UID: "a218303e-8766-4357-9cfe-e957492ea3b0") : secret "samples-operator-tls" not found
May 26 16:03:32.072631 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.072608    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-default-certificate\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.072725 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.072642    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-stats-auth\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.079413 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.079394    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk6hx\" (UniqueName: \"kubernetes.io/projected/b0ad9be8-9903-47e5-bc99-2c61845cce5e-kube-api-access-lk6hx\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.086454 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.086426    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2vr\" (UniqueName: \"kubernetes.io/projected/a218303e-8766-4357-9cfe-e957492ea3b0-kube-api-access-hj2vr\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:32.151746 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.151715    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-5nqbd"]
May 26 16:03:32.154615 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:03:32.154590    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe2ee2b_83d4_401a_a6e1_389caeeb7e09.slice/crio-dbde432bc607eaf682ae4bc5e78558ade40ba340067dba3d043f962625a40f62 WatchSource:0}: Error finding container dbde432bc607eaf682ae4bc5e78558ade40ba340067dba3d043f962625a40f62: Status 404 returned error can't find the container with id dbde432bc607eaf682ae4bc5e78558ade40ba340067dba3d043f962625a40f62
May 26 16:03:32.170743 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.170719    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c6kvh\" (UniqueName: \"kubernetes.io/projected/aa82f87f-2f4a-4cd6-98cd-c72f77130748-kube-api-access-c6kvh\") pod \"service-ca-operator-57b5b68b8-nvhtb\" (UID: \"aa82f87f-2f4a-4cd6-98cd-c72f77130748\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.170818 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.170770    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa82f87f-2f4a-4cd6-98cd-c72f77130748-serving-cert\") pod \"service-ca-operator-57b5b68b8-nvhtb\" (UID: \"aa82f87f-2f4a-4cd6-98cd-c72f77130748\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.170881 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.170835    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkxg\" (UniqueName: \"kubernetes.io/projected/ce52bd45-0e01-4b65-814b-294a61999d19-kube-api-access-gvkxg\") pod \"network-check-source-7f44b6dd87-xgtzm\" (UID: \"ce52bd45-0e01-4b65-814b-294a61999d19\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm"
May 26 16:03:32.170934 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.170905    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa82f87f-2f4a-4cd6-98cd-c72f77130748-config\") pod \"service-ca-operator-57b5b68b8-nvhtb\" (UID: \"aa82f87f-2f4a-4cd6-98cd-c72f77130748\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.171451 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.171433    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa82f87f-2f4a-4cd6-98cd-c72f77130748-config\") pod \"service-ca-operator-57b5b68b8-nvhtb\" (UID: \"aa82f87f-2f4a-4cd6-98cd-c72f77130748\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.172909 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.172889    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa82f87f-2f4a-4cd6-98cd-c72f77130748-serving-cert\") pod \"service-ca-operator-57b5b68b8-nvhtb\" (UID: \"aa82f87f-2f4a-4cd6-98cd-c72f77130748\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.178180 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.178160    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6kvh\" (UniqueName: \"kubernetes.io/projected/aa82f87f-2f4a-4cd6-98cd-c72f77130748-kube-api-access-c6kvh\") pod \"service-ca-operator-57b5b68b8-nvhtb\" (UID: \"aa82f87f-2f4a-4cd6-98cd-c72f77130748\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.178684 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.178668    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkxg\" (UniqueName: \"kubernetes.io/projected/ce52bd45-0e01-4b65-814b-294a61999d19-kube-api-access-gvkxg\") pod \"network-check-source-7f44b6dd87-xgtzm\" (UID: \"ce52bd45-0e01-4b65-814b-294a61999d19\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm"
May 26 16:03:32.231232 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.231208    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm"
May 26 16:03:32.237810 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.237790    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"
May 26 16:03:32.354340 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.354312    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm"]
May 26 16:03:32.357360 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:03:32.357334    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce52bd45_0e01_4b65_814b_294a61999d19.slice/crio-e409e805c3d8774be0e836aff8328eedf9f1577aea053ca801d164b1514fd6a2 WatchSource:0}: Error finding container e409e805c3d8774be0e836aff8328eedf9f1577aea053ca801d164b1514fd6a2: Status 404 returned error can't find the container with id e409e805c3d8774be0e836aff8328eedf9f1577aea053ca801d164b1514fd6a2
May 26 16:03:32.367440 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.367415    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb"]
May 26 16:03:32.370813 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:03:32.370790    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa82f87f_2f4a_4cd6_98cd_c72f77130748.slice/crio-53fcde072ada8843dcfef35abcd53d91c79f56491edcb980885c03bdeace3e7d WatchSource:0}: Error finding container 53fcde072ada8843dcfef35abcd53d91c79f56491edcb980885c03bdeace3e7d: Status 404 returned error can't find the container with id 53fcde072ada8843dcfef35abcd53d91c79f56491edcb980885c03bdeace3e7d
May 26 16:03:32.573662 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.573590    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.573818 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.573731    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:32.573818 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.573759    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:03:32.573975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.573766    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:32.573975 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.573840    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:33.57380939 +0000 UTC m=+164.955131406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : secret "router-metrics-certs-default" not found
May 26 16:03:32.573975 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.573861    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:03:32.573975 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.573893    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:33.573871372 +0000 UTC m=+164.955193390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : configmap references non-existent config key: service-ca.crt
May 26 16:03:32.573975 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:32.573919    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls podName:a218303e-8766-4357-9cfe-e957492ea3b0 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:33.573911658 +0000 UTC m=+164.955233660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls") pod "cluster-samples-operator-697587599-9cs29" (UID: "a218303e-8766-4357-9cfe-e957492ea3b0") : secret "samples-operator-tls" not found
May 26 16:03:32.733496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.733462    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm" event={"ID":"ce52bd45-0e01-4b65-814b-294a61999d19","Type":"ContainerStarted","Data":"3a5a21e108157883daf6853d6561f9074bbac0ec16c9a4a36681b8ecafabbeb3"}
May 26 16:03:32.733496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.733503    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm" event={"ID":"ce52bd45-0e01-4b65-814b-294a61999d19","Type":"ContainerStarted","Data":"e409e805c3d8774be0e836aff8328eedf9f1577aea053ca801d164b1514fd6a2"}
May 26 16:03:32.734550 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.734522    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb" event={"ID":"aa82f87f-2f4a-4cd6-98cd-c72f77130748","Type":"ContainerStarted","Data":"53fcde072ada8843dcfef35abcd53d91c79f56491edcb980885c03bdeace3e7d"}
May 26 16:03:32.735342 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.735323    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-5nqbd" event={"ID":"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09","Type":"ContainerStarted","Data":"dbde432bc607eaf682ae4bc5e78558ade40ba340067dba3d043f962625a40f62"}
May 26 16:03:32.748827 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:32.748790    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-xgtzm" podStartSLOduration=1.748777757 podStartE2EDuration="1.748777757s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:03:32.747973215 +0000 UTC m=+164.129295240" watchObservedRunningTime="2026-05-26 16:03:32.748777757 +0000 UTC m=+164.130099783"
May 26 16:03:33.583443 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:33.583400    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:33.583639 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:33.583456    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:33.583639 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:33.583548    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:33.583639 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:33.583593    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:35.583569706 +0000 UTC m=+166.964891722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : configmap references non-existent config key: service-ca.crt
May 26 16:03:33.583796 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:33.583656    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:03:33.583796 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:33.583661    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:03:33.583796 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:33.583711    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls podName:a218303e-8766-4357-9cfe-e957492ea3b0 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:35.583696425 +0000 UTC m=+166.965018444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls") pod "cluster-samples-operator-697587599-9cs29" (UID: "a218303e-8766-4357-9cfe-e957492ea3b0") : secret "samples-operator-tls" not found
May 26 16:03:33.583796 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:33.583729    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:35.583719505 +0000 UTC m=+166.965041513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : secret "router-metrics-certs-default" not found
May 26 16:03:34.741001 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:34.740961    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-5nqbd" event={"ID":"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09","Type":"ContainerStarted","Data":"cd06e39a5490314a986b6c26bfc49131ecdf370bc22c789e5d0047adfdd77143"}
May 26 16:03:34.742253 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:34.742224    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb" event={"ID":"aa82f87f-2f4a-4cd6-98cd-c72f77130748","Type":"ContainerStarted","Data":"1b9e4e3b2a603d5bc4f54ed261fdbd423eb24923346d1bc9d4536decc97843e0"}
May 26 16:03:34.759250 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:34.759132    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-5nqbd" podStartSLOduration=1.293693733 podStartE2EDuration="3.759114551s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="2026-05-26 16:03:32.156239694 +0000 UTC m=+163.537561696" lastFinishedPulling="2026-05-26 16:03:34.621660511 +0000 UTC m=+166.002982514" observedRunningTime="2026-05-26 16:03:34.75867255 +0000 UTC m=+166.139994578" watchObservedRunningTime="2026-05-26 16:03:34.759114551 +0000 UTC m=+166.140436578"
May 26 16:03:34.772958 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:34.772919    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb" podStartSLOduration=1.520166696 podStartE2EDuration="3.772907373s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="2026-05-26 16:03:32.372455727 +0000 UTC m=+163.753777734" lastFinishedPulling="2026-05-26 16:03:34.625196382 +0000 UTC m=+166.006518411" observedRunningTime="2026-05-26 16:03:34.772208782 +0000 UTC m=+166.153530809" watchObservedRunningTime="2026-05-26 16:03:34.772907373 +0000 UTC m=+166.154229398"
May 26 16:03:35.597738 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:35.597702    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:35.597943 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:35.597758    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:35.597943 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:35.597779    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:35.597943 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:35.597866    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:03:35.597943 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:35.597880    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:03:35.597943 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:35.597919    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:39.597897765 +0000 UTC m=+170.979219768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : configmap references non-existent config key: service-ca.crt
May 26 16:03:35.597943 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:35.597944    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls podName:a218303e-8766-4357-9cfe-e957492ea3b0 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:39.597934519 +0000 UTC m=+170.979256525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls") pod "cluster-samples-operator-697587599-9cs29" (UID: "a218303e-8766-4357-9cfe-e957492ea3b0") : secret "samples-operator-tls" not found
May 26 16:03:35.598249 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:35.597959    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:39.597950907 +0000 UTC m=+170.979272914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : secret "router-metrics-certs-default" not found
May 26 16:03:36.231958 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:36.231925    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:03:38.991170 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:38.991106    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qm4f5_e3f547af-2cf4-4821-ae28-e2087a814658/dns-node-resolver/0.log"
May 26 16:03:39.627682 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:39.627645    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:39.627844 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:39.627695    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:39.627844 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:39.627784    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:03:39.627844 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:39.627790    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:47.627777201 +0000 UTC m=+179.009099210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : configmap references non-existent config key: service-ca.crt
May 26 16:03:39.627954 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:39.627847    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:39.627954 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:39.627918    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls podName:a218303e-8766-4357-9cfe-e957492ea3b0 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:47.62790727 +0000 UTC m=+179.009229273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls") pod "cluster-samples-operator-697587599-9cs29" (UID: "a218303e-8766-4357-9cfe-e957492ea3b0") : secret "samples-operator-tls" not found
May 26 16:03:39.627954 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:39.627925    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:03:39.628048 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:39.627990    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:47.627975182 +0000 UTC m=+179.009297204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : secret "router-metrics-certs-default" not found
May 26 16:03:39.791073 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:39.791047    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mm79x_bda7246b-9f6c-4339-86e9-2a97c2370213/node-ca/0.log"
May 26 16:03:40.231195 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:40.231162    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:03:47.691033 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:47.690998    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:47.691445 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:47.691046    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:47.691445 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:47.691071    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:47.691445 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:03:47.691190    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle podName:b0ad9be8-9903-47e5-bc99-2c61845cce5e nodeName:}" failed. No retries permitted until 2026-05-26 16:04:03.691171786 +0000 UTC m=+195.072493793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle") pod "router-default-5f76bdfcbb-7twj9" (UID: "b0ad9be8-9903-47e5-bc99-2c61845cce5e") : configmap references non-existent config key: service-ca.crt
May 26 16:03:47.693470 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:47.693445    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a218303e-8766-4357-9cfe-e957492ea3b0-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9cs29\" (UID: \"a218303e-8766-4357-9cfe-e957492ea3b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:47.693567 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:47.693452    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad9be8-9903-47e5-bc99-2c61845cce5e-metrics-certs\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:03:47.740835 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:47.740795    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"
May 26 16:03:47.859075 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:47.859044    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29"]
May 26 16:03:48.770580 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:48.770539    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29" event={"ID":"a218303e-8766-4357-9cfe-e957492ea3b0","Type":"ContainerStarted","Data":"fcd20c1fca578f39fd6da774c4feebb07560c405e4a9d469846daee2c17f9c53"}
May 26 16:03:49.774421 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:49.774332    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29" event={"ID":"a218303e-8766-4357-9cfe-e957492ea3b0","Type":"ContainerStarted","Data":"5b209ed7b744e9f8b543eb7c8c25db8f5b81770b66761d5bef16dedc09363d97"}
May 26 16:03:49.774421 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:49.774378    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29" event={"ID":"a218303e-8766-4357-9cfe-e957492ea3b0","Type":"ContainerStarted","Data":"ed8e3e9b8f0da1f7b700fbda021bde4debcd3d435b9d2f702367dbc70486f799"}
May 26 16:03:49.788915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:03:49.788868    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9cs29" podStartSLOduration=17.214250221 podStartE2EDuration="18.788851602s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="2026-05-26 16:03:47.903000662 +0000 UTC m=+179.284322665" lastFinishedPulling="2026-05-26 16:03:49.477602043 +0000 UTC m=+180.858924046" observedRunningTime="2026-05-26 16:03:49.788546494 +0000 UTC m=+181.169868519" watchObservedRunningTime="2026-05-26 16:03:49.788851602 +0000 UTC m=+181.170173628"
May 26 16:04:00.360085 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.360043    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-xmq5t"]
May 26 16:04:00.365753 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.365728    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78b4b769ff-tfc2v"]
May 26 16:04:00.365880 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.365861    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-xmq5t"
May 26 16:04:00.368036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.368017    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 26 16:04:00.368157 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.368141    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 26 16:04:00.368204 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.368141    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pgcd6\""
May 26 16:04:00.368397 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.368376    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-564hz"]
May 26 16:04:00.368504 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.368491    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.370316 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.370297    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nv2cx\""
May 26 16:04:00.370412 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.370395    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 16:04:00.370465 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.370432    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 16:04:00.370713 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.370700    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 16:04:00.371317 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.371300    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.373708 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.373688    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 16:04:00.373805 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.373777    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 16:04:00.374002 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.373984    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vtbnj\""
May 26 16:04:00.375540 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.375520    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 16:04:00.378357 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.378339    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-xmq5t"]
May 26 16:04:00.379564 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.379545    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b4b769ff-tfc2v"]
May 26 16:04:00.394607 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.394584    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-564hz"]
May 26 16:04:00.489454 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489421    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzhk\" (UniqueName: \"kubernetes.io/projected/c3267091-97bf-4e59-b36b-ae7157a89f87-kube-api-access-wvzhk\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.489634 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489468    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-ca-trust-extracted\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.489634 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489487    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-bound-sa-token\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.489634 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489553    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3267091-97bf-4e59-b36b-ae7157a89f87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.489634 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489618    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-image-registry-private-configuration\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.489791 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489640    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-registry-tls\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.489791 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489661    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c3267091-97bf-4e59-b36b-ae7157a89f87-crio-socket\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.489791 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489710    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfmmk\" (UniqueName: \"kubernetes.io/projected/6f1529c1-8dd5-4612-9e01-8a453e28f70a-kube-api-access-nfmmk\") pod \"downloads-57bd96b888-xmq5t\" (UID: \"6f1529c1-8dd5-4612-9e01-8a453e28f70a\") " pod="openshift-console/downloads-57bd96b888-xmq5t"
May 26 16:04:00.489791 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489729    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3267091-97bf-4e59-b36b-ae7157a89f87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.489791 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-registry-certificates\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.489791 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489783    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-trusted-ca\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.489974 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489808    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstnh\" (UniqueName: \"kubernetes.io/projected/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-kube-api-access-wstnh\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.489974 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489861    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-installation-pull-secrets\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.489974 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.489899    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3267091-97bf-4e59-b36b-ae7157a89f87-data-volume\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.590218 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590186    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3267091-97bf-4e59-b36b-ae7157a89f87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.590398 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590239    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-image-registry-private-configuration\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.590398 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590264    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-registry-tls\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.590398 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590312    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c3267091-97bf-4e59-b36b-ae7157a89f87-crio-socket\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.590398 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590372    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nfmmk\" (UniqueName: \"kubernetes.io/projected/6f1529c1-8dd5-4612-9e01-8a453e28f70a-kube-api-access-nfmmk\") pod \"downloads-57bd96b888-xmq5t\" (UID: \"6f1529c1-8dd5-4612-9e01-8a453e28f70a\") " pod="openshift-console/downloads-57bd96b888-xmq5t"
May 26 16:04:00.590398 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590391    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c3267091-97bf-4e59-b36b-ae7157a89f87-crio-socket\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.590636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590404    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3267091-97bf-4e59-b36b-ae7157a89f87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.590636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590439    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-registry-certificates\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.590636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590458    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-trusted-ca\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.590636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590485    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wstnh\" (UniqueName: \"kubernetes.io/projected/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-kube-api-access-wstnh\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.590636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590517    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-installation-pull-secrets\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.590636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590551    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3267091-97bf-4e59-b36b-ae7157a89f87-data-volume\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.590636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590587    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzhk\" (UniqueName: \"kubernetes.io/projected/c3267091-97bf-4e59-b36b-ae7157a89f87-kube-api-access-wvzhk\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.590636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590625    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-ca-trust-extracted\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.591012 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590650    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-bound-sa-token\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.591012 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.590852    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3267091-97bf-4e59-b36b-ae7157a89f87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.591319 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.591273    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3267091-97bf-4e59-b36b-ae7157a89f87-data-volume\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.591614 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.591588    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-trusted-ca\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.591721 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.591687    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-ca-trust-extracted\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.591844 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.591824    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-registry-certificates\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.592943 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.592921    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-image-registry-private-configuration\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.593061 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.593045    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-registry-tls\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.593466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.593445    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-installation-pull-secrets\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.593549 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.593530    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3267091-97bf-4e59-b36b-ae7157a89f87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.598864 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.598841    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfmmk\" (UniqueName: \"kubernetes.io/projected/6f1529c1-8dd5-4612-9e01-8a453e28f70a-kube-api-access-nfmmk\") pod \"downloads-57bd96b888-xmq5t\" (UID: \"6f1529c1-8dd5-4612-9e01-8a453e28f70a\") " pod="openshift-console/downloads-57bd96b888-xmq5t"
May 26 16:04:00.599706 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.599683    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-bound-sa-token\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.600188 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.600169    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzhk\" (UniqueName: \"kubernetes.io/projected/c3267091-97bf-4e59-b36b-ae7157a89f87-kube-api-access-wvzhk\") pod \"insights-runtime-extractor-564hz\" (UID: \"c3267091-97bf-4e59-b36b-ae7157a89f87\") " pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.600490 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.600474    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstnh\" (UniqueName: \"kubernetes.io/projected/4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6-kube-api-access-wstnh\") pod \"image-registry-78b4b769ff-tfc2v\" (UID: \"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6\") " pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.676560 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.676536    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-xmq5t"
May 26 16:04:00.684295 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.684255    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:00.688931 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.688907    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-564hz"
May 26 16:04:00.823215 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.823182    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-xmq5t"]
May 26 16:04:00.825991 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:00.825955    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1529c1_8dd5_4612_9e01_8a453e28f70a.slice/crio-6632a7c2987b72c31ddecfa5944443ef36d5260fd2266b996bf60afaac153789 WatchSource:0}: Error finding container 6632a7c2987b72c31ddecfa5944443ef36d5260fd2266b996bf60afaac153789: Status 404 returned error can't find the container with id 6632a7c2987b72c31ddecfa5944443ef36d5260fd2266b996bf60afaac153789
May 26 16:04:00.840141 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.840116    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b4b769ff-tfc2v"]
May 26 16:04:00.843869 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:00.843839    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2cfe75_3d68_4a25_971d_0b9c4b2be4b6.slice/crio-88de66cd11b8a4e8f03b8c01b956c1f256f5bd6fda1143788112ec2c9089cd5a WatchSource:0}: Error finding container 88de66cd11b8a4e8f03b8c01b956c1f256f5bd6fda1143788112ec2c9089cd5a: Status 404 returned error can't find the container with id 88de66cd11b8a4e8f03b8c01b956c1f256f5bd6fda1143788112ec2c9089cd5a
May 26 16:04:00.852701 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:00.852676    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-564hz"]
May 26 16:04:00.855824 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:00.855784    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3267091_97bf_4e59_b36b_ae7157a89f87.slice/crio-b0f6a69d0fb04b76ff7cc5e4c4e7549dbee31d70bc25a25e09e9a3aad3e5d8a2 WatchSource:0}: Error finding container b0f6a69d0fb04b76ff7cc5e4c4e7549dbee31d70bc25a25e09e9a3aad3e5d8a2: Status 404 returned error can't find the container with id b0f6a69d0fb04b76ff7cc5e4c4e7549dbee31d70bc25a25e09e9a3aad3e5d8a2
May 26 16:04:01.803382 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:01.803343    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-564hz" event={"ID":"c3267091-97bf-4e59-b36b-ae7157a89f87","Type":"ContainerStarted","Data":"85af50faab211225967e8652a622197e12caaa4ddce9454f6a3b900206eb3580"}
May 26 16:04:01.803382 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:01.803386    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-564hz" event={"ID":"c3267091-97bf-4e59-b36b-ae7157a89f87","Type":"ContainerStarted","Data":"46751dc5d268767ed5a8a942cdba4f03810d7604f5b6b1b692ede84a20f9fa79"}
May 26 16:04:01.803872 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:01.803397    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-564hz" event={"ID":"c3267091-97bf-4e59-b36b-ae7157a89f87","Type":"ContainerStarted","Data":"b0f6a69d0fb04b76ff7cc5e4c4e7549dbee31d70bc25a25e09e9a3aad3e5d8a2"}
May 26 16:04:01.804722 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:01.804691    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v" event={"ID":"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6","Type":"ContainerStarted","Data":"81407c17430734a97a4e5fcc302808a3e418a69d298acd2fea45f942d84181be"}
May 26 16:04:01.804851 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:01.804729    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v" event={"ID":"4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6","Type":"ContainerStarted","Data":"88de66cd11b8a4e8f03b8c01b956c1f256f5bd6fda1143788112ec2c9089cd5a"}
May 26 16:04:01.804851 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:01.804801    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:01.805855 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:01.805832    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-xmq5t" event={"ID":"6f1529c1-8dd5-4612-9e01-8a453e28f70a","Type":"ContainerStarted","Data":"6632a7c2987b72c31ddecfa5944443ef36d5260fd2266b996bf60afaac153789"}
May 26 16:04:01.824146 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:01.824059    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v" podStartSLOduration=1.824046627 podStartE2EDuration="1.824046627s" podCreationTimestamp="2026-05-26 16:04:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:04:01.822868845 +0000 UTC m=+193.204190864" watchObservedRunningTime="2026-05-26 16:04:01.824046627 +0000 UTC m=+193.205368652"
May 26 16:04:03.717824 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:03.717777    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:04:03.718473 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:03.718452    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ad9be8-9903-47e5-bc99-2c61845cce5e-service-ca-bundle\") pod \"router-default-5f76bdfcbb-7twj9\" (UID: \"b0ad9be8-9903-47e5-bc99-2c61845cce5e\") " pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:04:03.813269 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:03.813234    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-564hz" event={"ID":"c3267091-97bf-4e59-b36b-ae7157a89f87","Type":"ContainerStarted","Data":"79412b2c3b9eb7e45277a5319920557db212b5000db594a65f2d2b24cc9559d4"}
May 26 16:04:03.833028 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:03.832979    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-564hz" podStartSLOduration=1.656941689 podStartE2EDuration="3.832965927s" podCreationTimestamp="2026-05-26 16:04:00 +0000 UTC" firstStartedPulling="2026-05-26 16:04:00.90201881 +0000 UTC m=+192.283340826" lastFinishedPulling="2026-05-26 16:04:03.078043059 +0000 UTC m=+194.459365064" observedRunningTime="2026-05-26 16:04:03.831045489 +0000 UTC m=+195.212367516" watchObservedRunningTime="2026-05-26 16:04:03.832965927 +0000 UTC m=+195.214287951"
May 26 16:04:03.948582 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:03.948545    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qn9n8\""
May 26 16:04:03.956505 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:03.956480    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:04:04.100149 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:04.100112    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f76bdfcbb-7twj9"]
May 26 16:04:04.103383 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:04.103356    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ad9be8_9903_47e5_bc99_2c61845cce5e.slice/crio-aa73c45acef7e4e987a8841fcebd9c1974856771d7da983f35e61db053bbdc83 WatchSource:0}: Error finding container aa73c45acef7e4e987a8841fcebd9c1974856771d7da983f35e61db053bbdc83: Status 404 returned error can't find the container with id aa73c45acef7e4e987a8841fcebd9c1974856771d7da983f35e61db053bbdc83
May 26 16:04:04.817960 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:04.817923    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f76bdfcbb-7twj9" event={"ID":"b0ad9be8-9903-47e5-bc99-2c61845cce5e","Type":"ContainerStarted","Data":"6c7b03501d1f4916ae1f9c2fd9b7e8b5ec33f1c010b3221dc9490900ada8c7ef"}
May 26 16:04:04.818429 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:04.817967    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f76bdfcbb-7twj9" event={"ID":"b0ad9be8-9903-47e5-bc99-2c61845cce5e","Type":"ContainerStarted","Data":"aa73c45acef7e4e987a8841fcebd9c1974856771d7da983f35e61db053bbdc83"}
May 26 16:04:04.840841 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:04.840778    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-5f76bdfcbb-7twj9" podStartSLOduration=33.84055743 podStartE2EDuration="33.84055743s" podCreationTimestamp="2026-05-26 16:03:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:04:04.839564119 +0000 UTC m=+196.220886157" watchObservedRunningTime="2026-05-26 16:04:04.84055743 +0000 UTC m=+196.221879456"
May 26 16:04:04.957403 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:04.957361    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:04:04.960383 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:04.960358    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:04:05.821253 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:05.821223    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:04:05.822633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:05.822608    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f76bdfcbb-7twj9"
May 26 16:04:08.161576 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.161540    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c488887cf-4p4dz"]
May 26 16:04:08.164505 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.164480    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.166791 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.166766    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 26 16:04:08.166919 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.166806    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 26 16:04:08.166986 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.166953    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xfnzp\""
May 26 16:04:08.167872 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.167848    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 26 16:04:08.167986 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.167871    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 26 16:04:08.168159 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.168140    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 26 16:04:08.175933 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.175911    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c488887cf-4p4dz"]
May 26 16:04:08.255790 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.255748    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-oauth-config\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.255957 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.255805    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vp9\" (UniqueName: \"kubernetes.io/projected/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-kube-api-access-r2vp9\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.255957 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.255883    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-config\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.255957 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.255923    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-serving-cert\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.256067 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.255985    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-oauth-serving-cert\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.256067 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.256015    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-service-ca\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.356686 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.356644    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-oauth-serving-cert\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.356868 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.356707    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-service-ca\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.356868 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.356767    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-oauth-config\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.356868 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.356820    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vp9\" (UniqueName: \"kubernetes.io/projected/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-kube-api-access-r2vp9\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.356868 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.356847    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-config\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.357073 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.356880    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-serving-cert\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.357546 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.357515    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-oauth-serving-cert\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.357546 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.357528    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-service-ca\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.357736 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.357534    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-config\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.359541 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.359520    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-oauth-config\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.359632 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.359572    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-serving-cert\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.364789 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.364762    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vp9\" (UniqueName: \"kubernetes.io/projected/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-kube-api-access-r2vp9\") pod \"console-7c488887cf-4p4dz\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") " pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.475726 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.475638    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:08.610595 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.610484    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c488887cf-4p4dz"]
May 26 16:04:08.613583 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:08.613543    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb190ce6f_c7c6_4f37_bd4d_cf541e6270ba.slice/crio-e70d96b260f272763d0fa4b9f7a0eea175884048cb6679602ce415db341b62da WatchSource:0}: Error finding container e70d96b260f272763d0fa4b9f7a0eea175884048cb6679602ce415db341b62da: Status 404 returned error can't find the container with id e70d96b260f272763d0fa4b9f7a0eea175884048cb6679602ce415db341b62da
May 26 16:04:08.829975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:08.829898    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c488887cf-4p4dz" event={"ID":"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba","Type":"ContainerStarted","Data":"e70d96b260f272763d0fa4b9f7a0eea175884048cb6679602ce415db341b62da"}
May 26 16:04:14.696124 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.696087    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"]
May 26 16:04:14.699756 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.699730    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rvcn8"]
May 26 16:04:14.699889 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.699836    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.702404 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.702379    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 26 16:04:14.702518 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.702485    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 16:04:14.702572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.702534    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6w5lw\""
May 26 16:04:14.702636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.702623    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 16:04:14.702738 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.702723    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 16:04:14.703055 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.703034    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.703556 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.703538    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 26 16:04:14.703640 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.703551    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 26 16:04:14.705034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.705017    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 16:04:14.705452 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.705432    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g5xd9\""
May 26 16:04:14.705528 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.705464    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 16:04:14.705582 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.705465    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 16:04:14.713609 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.713589    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"]
May 26 16:04:14.815606 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815564    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41f76d72-50c2-441c-9a2c-e2214d8fe34e-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.815776 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815618    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-wtmp\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.815776 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815683    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.815776 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815716    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-tls\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.815776 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815750    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-textfile\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.815947 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815786    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.815947 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815811    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tfd2\" (UniqueName: \"kubernetes.io/projected/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-kube-api-access-4tfd2\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.815947 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815837    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzmrn\" (UniqueName: \"kubernetes.io/projected/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-api-access-nzmrn\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.815947 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815866    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-sys\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.815947 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815932    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-root\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.816119 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.815976    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-metrics-client-ca\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.816119 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.816031    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41f76d72-50c2-441c-9a2c-e2214d8fe34e-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.816119 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.816089    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.816230 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.816135    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.816230 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.816173    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.916954 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.916909    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.916954 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.916951    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.917192 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917085    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.917255 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917238    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41f76d72-50c2-441c-9a2c-e2214d8fe34e-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.917320 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917293    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-wtmp\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917379 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917334    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917379 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917369    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-tls\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917475 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917411    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-textfile\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917475 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917458    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917575 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917482    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-wtmp\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917575 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917485    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfd2\" (UniqueName: \"kubernetes.io/projected/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-kube-api-access-4tfd2\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917575 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917566    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmrn\" (UniqueName: \"kubernetes.io/projected/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-api-access-nzmrn\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.917707 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917603    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41f76d72-50c2-441c-9a2c-e2214d8fe34e-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.917707 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917613    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-sys\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917707 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:14.917661    2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 16:04:14.917707 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917695    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-root\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917915 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:14.917727    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-tls podName:db87ceaf-fcba-4bfa-9e4d-3a448a6f9224 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:15.417708017 +0000 UTC m=+206.799030031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-tls") pod "node-exporter-rvcn8" (UID: "db87ceaf-fcba-4bfa-9e4d-3a448a6f9224") : secret "node-exporter-tls" not found
May 26 16:04:14.917915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917663    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-root\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917732    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-sys\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917740    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.917915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917765    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-metrics-client-ca\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917797    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-textfile\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.917915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.917877    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41f76d72-50c2-441c-9a2c-e2214d8fe34e-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.918340 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.918243    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.918340 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.918292    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-metrics-client-ca\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.918577 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.918557    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41f76d72-50c2-441c-9a2c-e2214d8fe34e-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.920127 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.920076    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.920254 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.920183    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.920344 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.920272    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:14.930354 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.930333    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tfd2\" (UniqueName: \"kubernetes.io/projected/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-kube-api-access-4tfd2\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:14.930550 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:14.930529    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmrn\" (UniqueName: \"kubernetes.io/projected/41f76d72-50c2-441c-9a2c-e2214d8fe34e-kube-api-access-nzmrn\") pod \"kube-state-metrics-54fd6bc5c4-dlzz7\" (UID: \"41f76d72-50c2-441c-9a2c-e2214d8fe34e\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:15.011608 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.011518    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"
May 26 16:04:15.423400 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.423367    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-tls\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:15.423582 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:15.423521    2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 16:04:15.423639 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:15.423607    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-tls podName:db87ceaf-fcba-4bfa-9e4d-3a448a6f9224 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:16.423586598 +0000 UTC m=+207.804908621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-tls") pod "node-exporter-rvcn8" (UID: "db87ceaf-fcba-4bfa-9e4d-3a448a6f9224") : secret "node-exporter-tls" not found
May 26 16:04:15.807910 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.807826    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:04:15.814329 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.814296    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.817036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.816823    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 26 16:04:15.817036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.816895    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 26 16:04:15.817036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.816832    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 26 16:04:15.817036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.816923    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 26 16:04:15.817349 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.817170    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 26 16:04:15.817349 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.817204    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 26 16:04:15.817349 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.817246    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 26 16:04:15.817504 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.817415    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 26 16:04:15.817745 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.817724    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xbwld\""
May 26 16:04:15.822429 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.822405    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 26 16:04:15.823649 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.823625    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:04:15.927758 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.927723    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.927925 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.927771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-out\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.927925 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.927858    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-volume\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.927925 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.927888    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.928064 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.927927    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.928064 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.927952    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285xd\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-kube-api-access-285xd\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.928064 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.927975    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.928064 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.927993    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-web-config\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.928064 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.928021    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.928255 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.928148    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.928255 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.928201    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.928255 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.928227    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:15.928372 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:15.928289    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.029778 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.029738    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.029938 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.029799    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.029938 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.029830    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-out\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.029938 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.029873    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-volume\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.029938 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.029902    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.030131 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.030093    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.030179 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.030136    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-285xd\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-kube-api-access-285xd\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.030229 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.030179    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.030229 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.030209    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-web-config\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.030355 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.030245    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.030410 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.030365    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.030457 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.030425    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.030457 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.030449    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.031350 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.031321    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.031951 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.031532    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.031951 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.031587    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.031951 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:16.031636    2570 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 26 16:04:16.031951 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:16.031716    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-main-tls podName:146de6bf-dc8f-46fa-bd57-1f1647d543fa nodeName:}" failed. No retries permitted until 2026-05-26 16:04:16.531695863 +0000 UTC m=+207.913017890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa") : secret "alertmanager-main-tls" not found
May 26 16:04:16.032816 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.032680    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-out\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.032936 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.032851    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.033601 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.033576    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.034131 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.034112    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.034212 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.034152    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-volume\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.034767 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.034735    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.034862 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.034811    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.035138 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.035117    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-web-config\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.039368 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.039344    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-285xd\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-kube-api-access-285xd\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.433358 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.433324    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-tls\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:16.435501 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.435470    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db87ceaf-fcba-4bfa-9e4d-3a448a6f9224-node-exporter-tls\") pod \"node-exporter-rvcn8\" (UID: \"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224\") " pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:16.518460 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.518418    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvcn8"
May 26 16:04:16.534624 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.534594    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.537154 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.537128    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:16.727038 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:16.726948    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:17.654214 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.654186    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:04:17.655195 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:17.655169    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146de6bf_dc8f_46fa_bd57_1f1647d543fa.slice/crio-f989060d8d208c1ad3abdf7b5c362ef74d5332bc0dbbf857b1083c9b08269608 WatchSource:0}: Error finding container f989060d8d208c1ad3abdf7b5c362ef74d5332bc0dbbf857b1083c9b08269608: Status 404 returned error can't find the container with id f989060d8d208c1ad3abdf7b5c362ef74d5332bc0dbbf857b1083c9b08269608
May 26 16:04:17.668056 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.668030    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7"]
May 26 16:04:17.671694 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:17.671671    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f76d72_50c2_441c_9a2c_e2214d8fe34e.slice/crio-d4f35ba236de3f348dd4fd7fe5b361bbe9d11dc21271169fbd7a8517707a6584 WatchSource:0}: Error finding container d4f35ba236de3f348dd4fd7fe5b361bbe9d11dc21271169fbd7a8517707a6584: Status 404 returned error can't find the container with id d4f35ba236de3f348dd4fd7fe5b361bbe9d11dc21271169fbd7a8517707a6584
May 26 16:04:17.857194 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.857103    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-xmq5t" event={"ID":"6f1529c1-8dd5-4612-9e01-8a453e28f70a","Type":"ContainerStarted","Data":"ba27d95c3b1d3770c1ef1a2e524f04d85c7af05da4332a4e1256cddcdecad59b"}
May 26 16:04:17.857426 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.857404    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-xmq5t"
May 26 16:04:17.858947 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.858899    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c488887cf-4p4dz" event={"ID":"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba","Type":"ContainerStarted","Data":"3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230"}
May 26 16:04:17.860157 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.860129    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerStarted","Data":"f989060d8d208c1ad3abdf7b5c362ef74d5332bc0dbbf857b1083c9b08269608"}
May 26 16:04:17.861312 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.861271    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvcn8" event={"ID":"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224","Type":"ContainerStarted","Data":"e843700b5839e48020cf1b17fc3ee322fa796b5efee8d41fd35d66926d78ca4a"}
May 26 16:04:17.862511 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.862485    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7" event={"ID":"41f76d72-50c2-441c-9a2c-e2214d8fe34e","Type":"ContainerStarted","Data":"d4f35ba236de3f348dd4fd7fe5b361bbe9d11dc21271169fbd7a8517707a6584"}
May 26 16:04:17.874060 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.874013    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-xmq5t" podStartSLOduration=1.154112918 podStartE2EDuration="17.873998559s" podCreationTimestamp="2026-05-26 16:04:00 +0000 UTC" firstStartedPulling="2026-05-26 16:04:00.827828402 +0000 UTC m=+192.209150408" lastFinishedPulling="2026-05-26 16:04:17.547714046 +0000 UTC m=+208.929036049" observedRunningTime="2026-05-26 16:04:17.872940722 +0000 UTC m=+209.254262757" watchObservedRunningTime="2026-05-26 16:04:17.873998559 +0000 UTC m=+209.255320584"
May 26 16:04:17.875579 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.875552    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-xmq5t"
May 26 16:04:17.890502 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:17.890460    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7c488887cf-4p4dz" podStartSLOduration=0.994687177 podStartE2EDuration="9.890448669s" podCreationTimestamp="2026-05-26 16:04:08 +0000 UTC" firstStartedPulling="2026-05-26 16:04:08.615687608 +0000 UTC m=+199.997009612" lastFinishedPulling="2026-05-26 16:04:17.511449087 +0000 UTC m=+208.892771104" observedRunningTime="2026-05-26 16:04:17.889432174 +0000 UTC m=+209.270754198" watchObservedRunningTime="2026-05-26 16:04:17.890448669 +0000 UTC m=+209.271770694"
May 26 16:04:18.476668 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:18.476563    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:18.476668 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:18.476644    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:18.478350 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:18.477964    2570 patch_prober.go:28] interesting pod/console-7c488887cf-4p4dz container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" start-of-body=
May 26 16:04:18.478350 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:18.478025    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7c488887cf-4p4dz" podUID="b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" containerName="console" probeResult="failure" output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused"
May 26 16:04:19.070345 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.070309    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7888fb4b49-lfbg2"]
May 26 16:04:19.074305 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.074269    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.078079 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.077751    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 26 16:04:19.078079 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.077817    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 26 16:04:19.078079 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.077752    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 26 16:04:19.078079 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.077752    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 26 16:04:19.078079 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.077765    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4zbdf\""
May 26 16:04:19.078079 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.077823    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b16clot7qfare\""
May 26 16:04:19.083961 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.083513    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7888fb4b49-lfbg2"]
May 26 16:04:19.163356 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.163309    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnlr\" (UniqueName: \"kubernetes.io/projected/bc101f86-414b-4376-9851-b0668457cdd8-kube-api-access-zdnlr\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.163535 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.163498    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bc101f86-414b-4376-9851-b0668457cdd8-client-ca-bundle\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.163601 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.163544    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bc101f86-414b-4376-9851-b0668457cdd8-metrics-server-audit-profiles\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.163601 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.163579    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bc101f86-414b-4376-9851-b0668457cdd8-secret-metrics-server-client-certs\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.163704 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.163604    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bc101f86-414b-4376-9851-b0668457cdd8-secret-metrics-server-tls\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.163704 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.163639    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc101f86-414b-4376-9851-b0668457cdd8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.163704 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.163697    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bc101f86-414b-4376-9851-b0668457cdd8-audit-log\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.264409 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.264371    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bc101f86-414b-4376-9851-b0668457cdd8-client-ca-bundle\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.264633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.264431    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bc101f86-414b-4376-9851-b0668457cdd8-metrics-server-audit-profiles\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.264633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.264472    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bc101f86-414b-4376-9851-b0668457cdd8-secret-metrics-server-client-certs\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.264633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.264501    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bc101f86-414b-4376-9851-b0668457cdd8-secret-metrics-server-tls\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.264633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.264545    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc101f86-414b-4376-9851-b0668457cdd8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.264633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.264582    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bc101f86-414b-4376-9851-b0668457cdd8-audit-log\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.264633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.264619    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnlr\" (UniqueName: \"kubernetes.io/projected/bc101f86-414b-4376-9851-b0668457cdd8-kube-api-access-zdnlr\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.266347 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.265993    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bc101f86-414b-4376-9851-b0668457cdd8-audit-log\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.266347 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.266252    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc101f86-414b-4376-9851-b0668457cdd8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.268208 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.267944    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bc101f86-414b-4376-9851-b0668457cdd8-secret-metrics-server-tls\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.269062 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.268954    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bc101f86-414b-4376-9851-b0668457cdd8-secret-metrics-server-client-certs\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.269367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.269342    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bc101f86-414b-4376-9851-b0668457cdd8-metrics-server-audit-profiles\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.270548 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.270491    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bc101f86-414b-4376-9851-b0668457cdd8-client-ca-bundle\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.273641 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.273601    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnlr\" (UniqueName: \"kubernetes.io/projected/bc101f86-414b-4376-9851-b0668457cdd8-kube-api-access-zdnlr\") pod \"metrics-server-7888fb4b49-lfbg2\" (UID: \"bc101f86-414b-4376-9851-b0668457cdd8\") " pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.387664 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.387577    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:19.450564 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.450534    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"]
May 26 16:04:19.476134 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.476096    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"]
May 26 16:04:19.476323 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.476250    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"
May 26 16:04:19.478635 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.478607    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 26 16:04:19.478755 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.478643    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-6z4pq\""
May 26 16:04:19.567263 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.567230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66bc4d8e-4730-4cc9-97d5-48fb27980a67-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-tqjc9\" (UID: \"66bc4d8e-4730-4cc9-97d5-48fb27980a67\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"
May 26 16:04:19.668086 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.668006    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66bc4d8e-4730-4cc9-97d5-48fb27980a67-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-tqjc9\" (UID: \"66bc4d8e-4730-4cc9-97d5-48fb27980a67\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"
May 26 16:04:19.668240 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:19.668177    2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 26 16:04:19.668315 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:19.668258    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bc4d8e-4730-4cc9-97d5-48fb27980a67-monitoring-plugin-cert podName:66bc4d8e-4730-4cc9-97d5-48fb27980a67 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:20.168236483 +0000 UTC m=+211.549558511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/66bc4d8e-4730-4cc9-97d5-48fb27980a67-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-tqjc9" (UID: "66bc4d8e-4730-4cc9-97d5-48fb27980a67") : secret "monitoring-plugin-cert" not found
May 26 16:04:19.884643 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.884607    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"]
May 26 16:04:19.906207 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.906177    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"]
May 26 16:04:19.906388 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.906336    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:19.908820 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.908795    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 26 16:04:19.909043 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.909021    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 26 16:04:19.909150 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.909055    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 26 16:04:19.909150 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.909135    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 26 16:04:19.909300 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.909175    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9vkpd\""
May 26 16:04:19.909487 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.909467    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 26 16:04:19.914010 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:19.913989    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 26 16:04:20.072672 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.072572    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-telemeter-client-tls\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.072672 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.072677    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-federate-client-tls\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.073367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.072713    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg8ng\" (UniqueName: \"kubernetes.io/projected/a07153bc-afe6-4068-bec7-b741121f5c09-kube-api-access-rg8ng\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.073367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.072771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07153bc-afe6-4068-bec7-b741121f5c09-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.073367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.072804    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07153bc-afe6-4068-bec7-b741121f5c09-serving-certs-ca-bundle\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.073367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.072846    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.073367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.072876    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a07153bc-afe6-4068-bec7-b741121f5c09-metrics-client-ca\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.073367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.072920    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-secret-telemeter-client\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.127562 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.127200    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7888fb4b49-lfbg2"]
May 26 16:04:20.131844 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:20.131792    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc101f86_414b_4376_9851_b0668457cdd8.slice/crio-5dda4dff506f8dc3130c6ab32fa7617a2573bb4af4aac63dcf3c409807acef4e WatchSource:0}: Error finding container 5dda4dff506f8dc3130c6ab32fa7617a2573bb4af4aac63dcf3c409807acef4e: Status 404 returned error can't find the container with id 5dda4dff506f8dc3130c6ab32fa7617a2573bb4af4aac63dcf3c409807acef4e
May 26 16:04:20.175042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.174332    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-telemeter-client-tls\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.175042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.174401    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66bc4d8e-4730-4cc9-97d5-48fb27980a67-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-tqjc9\" (UID: \"66bc4d8e-4730-4cc9-97d5-48fb27980a67\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"
May 26 16:04:20.175042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.174432    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-federate-client-tls\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.175042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.174461    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rg8ng\" (UniqueName: \"kubernetes.io/projected/a07153bc-afe6-4068-bec7-b741121f5c09-kube-api-access-rg8ng\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.175042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.174518    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07153bc-afe6-4068-bec7-b741121f5c09-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.175042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.174551    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07153bc-afe6-4068-bec7-b741121f5c09-serving-certs-ca-bundle\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.175042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.174595    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.175042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.174626    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a07153bc-afe6-4068-bec7-b741121f5c09-metrics-client-ca\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.175042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.174671    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-secret-telemeter-client\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.178903 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.175891    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07153bc-afe6-4068-bec7-b741121f5c09-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.178903 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.176307    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a07153bc-afe6-4068-bec7-b741121f5c09-metrics-client-ca\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.178903 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.176859    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07153bc-afe6-4068-bec7-b741121f5c09-serving-certs-ca-bundle\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.178903 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.178566    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-federate-client-tls\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.180105 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.179642    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.180105 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.179673    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-telemeter-client-tls\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.180105 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.179817    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66bc4d8e-4730-4cc9-97d5-48fb27980a67-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-tqjc9\" (UID: \"66bc4d8e-4730-4cc9-97d5-48fb27980a67\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"
May 26 16:04:20.180105 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.179897    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a07153bc-afe6-4068-bec7-b741121f5c09-secret-telemeter-client\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.185412 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.184748    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg8ng\" (UniqueName: \"kubernetes.io/projected/a07153bc-afe6-4068-bec7-b741121f5c09-kube-api-access-rg8ng\") pod \"telemeter-client-6c4b68ff49-d95c8\" (UID: \"a07153bc-afe6-4068-bec7-b741121f5c09\") " pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.218103 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.217989    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"
May 26 16:04:20.387865 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.387812    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"
May 26 16:04:20.473429 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.465688    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c4b68ff49-d95c8"]
May 26 16:04:20.478266 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:20.477963    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07153bc_afe6_4068_bec7_b741121f5c09.slice/crio-92408948b4f0d1e4f0fb7b6af1b719c641a892db402df436bebcc2cb62d00124 WatchSource:0}: Error finding container 92408948b4f0d1e4f0fb7b6af1b719c641a892db402df436bebcc2cb62d00124: Status 404 returned error can't find the container with id 92408948b4f0d1e4f0fb7b6af1b719c641a892db402df436bebcc2cb62d00124
May 26 16:04:20.561435 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.561307    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"]
May 26 16:04:20.564477 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:20.564445    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bc4d8e_4730_4cc9_97d5_48fb27980a67.slice/crio-afb87570c0858764c2c3bac2cabe6f3e192db1041fe54c61e658cff7b22c67f9 WatchSource:0}: Error finding container afb87570c0858764c2c3bac2cabe6f3e192db1041fe54c61e658cff7b22c67f9: Status 404 returned error can't find the container with id afb87570c0858764c2c3bac2cabe6f3e192db1041fe54c61e658cff7b22c67f9
May 26 16:04:20.688712 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.688677    2570 patch_prober.go:28] interesting pod/image-registry-78b4b769ff-tfc2v container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:04:20.688890 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.688731    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v" podUID="4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:04:20.874412 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.874214    2570 generic.go:358] "Generic (PLEG): container finished" podID="db87ceaf-fcba-4bfa-9e4d-3a448a6f9224" containerID="9197303e23e5067b8737984290376f36a002a8b231869210eafa789ed6af720a" exitCode=0
May 26 16:04:20.874412 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.874327    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvcn8" event={"ID":"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224","Type":"ContainerDied","Data":"9197303e23e5067b8737984290376f36a002a8b231869210eafa789ed6af720a"}
May 26 16:04:20.877501 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.877454    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9" event={"ID":"66bc4d8e-4730-4cc9-97d5-48fb27980a67","Type":"ContainerStarted","Data":"afb87570c0858764c2c3bac2cabe6f3e192db1041fe54c61e658cff7b22c67f9"}
May 26 16:04:20.879865 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.879835    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7" event={"ID":"41f76d72-50c2-441c-9a2c-e2214d8fe34e","Type":"ContainerStarted","Data":"d30c3a3bf7f3114ce180ded17c7398ac126fee90d063f1d11bac230f85a546d5"}
May 26 16:04:20.880001 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.879872    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7" event={"ID":"41f76d72-50c2-441c-9a2c-e2214d8fe34e","Type":"ContainerStarted","Data":"09084fb0b864d786e21c8fec4b71f60642f38e19d8daf1df28b45fcc6d8fe060"}
May 26 16:04:20.880001 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.879886    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7" event={"ID":"41f76d72-50c2-441c-9a2c-e2214d8fe34e","Type":"ContainerStarted","Data":"de34141348dfd7e2170f9b00c0c40f4a797ee2cdac175c9d3640ca75afd4a5cd"}
May 26 16:04:20.883209 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.883161    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2" event={"ID":"bc101f86-414b-4376-9851-b0668457cdd8","Type":"ContainerStarted","Data":"5dda4dff506f8dc3130c6ab32fa7617a2573bb4af4aac63dcf3c409807acef4e"}
May 26 16:04:20.885175 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.885067    2570 generic.go:358] "Generic (PLEG): container finished" podID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerID="511adab247fd48261fbd24cb45a7dd804f8ca70fa447ebf0a2dc12e01ff276e4" exitCode=0
May 26 16:04:20.885175 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.885147    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerDied","Data":"511adab247fd48261fbd24cb45a7dd804f8ca70fa447ebf0a2dc12e01ff276e4"}
May 26 16:04:20.886634 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.886610    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8" event={"ID":"a07153bc-afe6-4068-bec7-b741121f5c09","Type":"ContainerStarted","Data":"92408948b4f0d1e4f0fb7b6af1b719c641a892db402df436bebcc2cb62d00124"}
May 26 16:04:20.946589 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:20.945381    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dlzz7" podStartSLOduration=4.639686972 podStartE2EDuration="6.945360043s" podCreationTimestamp="2026-05-26 16:04:14 +0000 UTC" firstStartedPulling="2026-05-26 16:04:17.673884687 +0000 UTC m=+209.055206694" lastFinishedPulling="2026-05-26 16:04:19.979557747 +0000 UTC m=+211.360879765" observedRunningTime="2026-05-26 16:04:20.941880927 +0000 UTC m=+212.323202953" watchObservedRunningTime="2026-05-26 16:04:20.945360043 +0000 UTC m=+212.326682070"
May 26 16:04:21.894888 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:21.894845    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvcn8" event={"ID":"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224","Type":"ContainerStarted","Data":"22a07fc44a88a14f95515d7966b049db9620a150425afab59981c7d2fd4407a3"}
May 26 16:04:21.894888 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:21.894893    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvcn8" event={"ID":"db87ceaf-fcba-4bfa-9e4d-3a448a6f9224","Type":"ContainerStarted","Data":"17298d271a30e8adf2e134f31c8fe406460d59153a0deb9f103bf43a4463d3b0"}
May 26 16:04:21.917733 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:21.917668    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rvcn8" podStartSLOduration=5.448537531 podStartE2EDuration="7.917637061s" podCreationTimestamp="2026-05-26 16:04:14 +0000 UTC" firstStartedPulling="2026-05-26 16:04:17.505428724 +0000 UTC m=+208.886750740" lastFinishedPulling="2026-05-26 16:04:19.974528263 +0000 UTC m=+211.355850270" observedRunningTime="2026-05-26 16:04:21.91505734 +0000 UTC m=+213.296379392" watchObservedRunningTime="2026-05-26 16:04:21.917637061 +0000 UTC m=+213.298959087"
May 26 16:04:22.813227 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:22.813196    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78b4b769ff-tfc2v"
May 26 16:04:24.907235 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:24.907181    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerStarted","Data":"567894c1b53c77cdb146e7d502e9b90cdeb35970a5575dd4a71a8c26818c1274"}
May 26 16:04:24.908761 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:24.908725    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8" event={"ID":"a07153bc-afe6-4068-bec7-b741121f5c09","Type":"ContainerStarted","Data":"4d97e613ec65b7a40268aee03512e77a49afd4c4e7e8c169ffc10634f151d4a4"}
May 26 16:04:24.912664 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:24.910364    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9" event={"ID":"66bc4d8e-4730-4cc9-97d5-48fb27980a67","Type":"ContainerStarted","Data":"fe7be3d63408c9b476241fe345501432e1fbb4b5febf4dc389d1e4d3fbecd9e7"}
May 26 16:04:24.913304 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:24.913189    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"
May 26 16:04:24.919869 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:24.919301    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2" event={"ID":"bc101f86-414b-4376-9851-b0668457cdd8","Type":"ContainerStarted","Data":"a89501b0401468aa292fc49d817799f6f6962e56fca5eb2425ba5b94268d763a"}
May 26 16:04:24.923700 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:24.923660    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9"
May 26 16:04:24.941966 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:24.941787    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-tqjc9" podStartSLOduration=1.829826527 podStartE2EDuration="5.941768563s" podCreationTimestamp="2026-05-26 16:04:19 +0000 UTC" firstStartedPulling="2026-05-26 16:04:20.566879832 +0000 UTC m=+211.948201837" lastFinishedPulling="2026-05-26 16:04:24.678821855 +0000 UTC m=+216.060143873" observedRunningTime="2026-05-26 16:04:24.926619426 +0000 UTC m=+216.307941473" watchObservedRunningTime="2026-05-26 16:04:24.941768563 +0000 UTC m=+216.323090589"
May 26 16:04:24.963176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:24.963131    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2" podStartSLOduration=1.417941001 podStartE2EDuration="5.963116504s" podCreationTimestamp="2026-05-26 16:04:19 +0000 UTC" firstStartedPulling="2026-05-26 16:04:20.134687982 +0000 UTC m=+211.516009999" lastFinishedPulling="2026-05-26 16:04:24.679863492 +0000 UTC m=+216.061185502" observedRunningTime="2026-05-26 16:04:24.9614267 +0000 UTC m=+216.342748726" watchObservedRunningTime="2026-05-26 16:04:24.963116504 +0000 UTC m=+216.344438530"
May 26 16:04:25.560758 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.560676    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c488887cf-4p4dz"]
May 26 16:04:25.600591 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.600561    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-9bdd488d-t42v4"]
May 26 16:04:25.624581 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.624544    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bdd488d-t42v4"]
May 26 16:04:25.624756 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.624683    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.631745 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.631711    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 26 16:04:25.640377 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.640036    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-trusted-ca-bundle\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.640377 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.640079    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-serving-cert\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.640377 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.640128    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-console-config\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.640377 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.640164    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-service-ca\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.640377 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.640217    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5j4p\" (UniqueName: \"kubernetes.io/projected/d86a048e-fd12-4324-a820-ee27cf6704a3-kube-api-access-h5j4p\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.640377 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.640246    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-oauth-config\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.640377 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.640272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-oauth-serving-cert\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.741749 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.741134    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-service-ca\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.741749 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.741206    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h5j4p\" (UniqueName: \"kubernetes.io/projected/d86a048e-fd12-4324-a820-ee27cf6704a3-kube-api-access-h5j4p\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.741749 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.741242    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-oauth-config\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.741749 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.741269    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-oauth-serving-cert\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.741749 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.741373    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-trusted-ca-bundle\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.741749 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.741407    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-serving-cert\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.741749 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.741467    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-console-config\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.742250 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.741822    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-service-ca\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.742684 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.742624    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-oauth-serving-cert\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.742792 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.742727    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-console-config\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.742872 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.742844    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-trusted-ca-bundle\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.744584 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.744558    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-serving-cert\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.744971 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.744951    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-oauth-config\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.749769 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.749745    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5j4p\" (UniqueName: \"kubernetes.io/projected/d86a048e-fd12-4324-a820-ee27cf6704a3-kube-api-access-h5j4p\") pod \"console-9bdd488d-t42v4\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") " pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.926508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.926408    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerStarted","Data":"551232802f169c3f4965f042d052ba2c540c85d68136b087175a202611b08a5c"}
May 26 16:04:25.926508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.926456    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerStarted","Data":"8c8a673d0910dd4a859cc51c8c10a394ffb8d9417df12d25cf615a20337634ff"}
May 26 16:04:25.926508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.926472    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerStarted","Data":"919ff67912e4c7f265c2ade9ebdadd33bb576980a6446b77c1a388a0deb4e97d"}
May 26 16:04:25.926508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.926486    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerStarted","Data":"2e0d0d5bce3ae419dcb0e353c28475f881fe52e141b536575108910f00432832"}
May 26 16:04:25.928789 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.928729    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8" event={"ID":"a07153bc-afe6-4068-bec7-b741121f5c09","Type":"ContainerStarted","Data":"6b84a20f8db2cc56ba52bc053237875cee6d7e7f87ba96632fae29147382d9f0"}
May 26 16:04:25.928789 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.928768    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8" event={"ID":"a07153bc-afe6-4068-bec7-b741121f5c09","Type":"ContainerStarted","Data":"a8c3245e8914766785c51d7d4351f6446917dca7989a19703c6e8a15b840a0a4"}
May 26 16:04:25.936422 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.936398    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:25.949020 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:25.948969    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c4b68ff49-d95c8" podStartSLOduration=2.749672425 podStartE2EDuration="6.948953648s" podCreationTimestamp="2026-05-26 16:04:19 +0000 UTC" firstStartedPulling="2026-05-26 16:04:20.481684568 +0000 UTC m=+211.863006586" lastFinishedPulling="2026-05-26 16:04:24.680965801 +0000 UTC m=+216.062287809" observedRunningTime="2026-05-26 16:04:25.946980287 +0000 UTC m=+217.328302316" watchObservedRunningTime="2026-05-26 16:04:25.948953648 +0000 UTC m=+217.330275675"
May 26 16:04:26.099395 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:26.099299    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bdd488d-t42v4"]
May 26 16:04:26.102625 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:26.102584    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86a048e_fd12_4324_a820_ee27cf6704a3.slice/crio-9b281d8050fa301a6d5b377c70edde0efd95d5e931d2ce95c3002f095d93349d WatchSource:0}: Error finding container 9b281d8050fa301a6d5b377c70edde0efd95d5e931d2ce95c3002f095d93349d: Status 404 returned error can't find the container with id 9b281d8050fa301a6d5b377c70edde0efd95d5e931d2ce95c3002f095d93349d
May 26 16:04:26.933180 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:26.933144    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bdd488d-t42v4" event={"ID":"d86a048e-fd12-4324-a820-ee27cf6704a3","Type":"ContainerStarted","Data":"2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a"}
May 26 16:04:26.933180 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:26.933183    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bdd488d-t42v4" event={"ID":"d86a048e-fd12-4324-a820-ee27cf6704a3","Type":"ContainerStarted","Data":"9b281d8050fa301a6d5b377c70edde0efd95d5e931d2ce95c3002f095d93349d"}
May 26 16:04:26.936012 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:26.935985    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerStarted","Data":"9758421cc20ac998332d48ff7a2d42217cdae0cf306e7a627d9e234c04ce723f"}
May 26 16:04:26.949844 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:26.949791    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-9bdd488d-t42v4" podStartSLOduration=1.949780467 podStartE2EDuration="1.949780467s" podCreationTimestamp="2026-05-26 16:04:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:04:26.948983104 +0000 UTC m=+218.330305130" watchObservedRunningTime="2026-05-26 16:04:26.949780467 +0000 UTC m=+218.331102491"
May 26 16:04:26.974514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:26.974473    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.90251727 podStartE2EDuration="11.974462632s" podCreationTimestamp="2026-05-26 16:04:15 +0000 UTC" firstStartedPulling="2026-05-26 16:04:17.65704183 +0000 UTC m=+209.038363837" lastFinishedPulling="2026-05-26 16:04:26.728987196 +0000 UTC m=+218.110309199" observedRunningTime="2026-05-26 16:04:26.972021443 +0000 UTC m=+218.353343562" watchObservedRunningTime="2026-05-26 16:04:26.974462632 +0000 UTC m=+218.355784657"
May 26 16:04:30.462328 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.462290    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bdd488d-t42v4"]
May 26 16:04:30.495875 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.495845    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-8569578f86-n87h2"]
May 26 16:04:30.522604 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.522575    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8569578f86-n87h2"]
May 26 16:04:30.522773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.522718    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.585608 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.585572    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-console-config\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.585771 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.585640    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-oauth-config\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.585771 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.585678    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-oauth-serving-cert\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.585842 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.585771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-trusted-ca-bundle\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.585842 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.585829    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5px\" (UniqueName: \"kubernetes.io/projected/9b75006f-f808-4e0e-b478-e06c266b3322-kube-api-access-fn5px\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.585910 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.585864    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-serving-cert\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.585910 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.585882    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-service-ca\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.687093 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.687056    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-oauth-serving-cert\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.687299 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.687137    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-trusted-ca-bundle\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.687299 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.687162    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5px\" (UniqueName: \"kubernetes.io/projected/9b75006f-f808-4e0e-b478-e06c266b3322-kube-api-access-fn5px\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.687299 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.687197    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-serving-cert\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.687299 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.687221    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-service-ca\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.687299 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.687269    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-console-config\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.687607 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.687361    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-oauth-config\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.688057 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.688029    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-service-ca\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.688176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.688078    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-console-config\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.688176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.688093    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-trusted-ca-bundle\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.688176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.688161    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-oauth-serving-cert\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.689929 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.689905    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-serving-cert\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.689929 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.689901    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-oauth-config\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.701991 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.701962    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5px\" (UniqueName: \"kubernetes.io/projected/9b75006f-f808-4e0e-b478-e06c266b3322-kube-api-access-fn5px\") pod \"console-8569578f86-n87h2\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") " pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.831486 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.831398    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:30.957772 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:30.957718    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8569578f86-n87h2"]
May 26 16:04:30.960304 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:04:30.960254    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b75006f_f808_4e0e_b478_e06c266b3322.slice/crio-cb51e6002161f45ae14d19b4d36e2e2caada39753333d87a70eb16ca9c752823 WatchSource:0}: Error finding container cb51e6002161f45ae14d19b4d36e2e2caada39753333d87a70eb16ca9c752823: Status 404 returned error can't find the container with id cb51e6002161f45ae14d19b4d36e2e2caada39753333d87a70eb16ca9c752823
May 26 16:04:31.950548 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:31.950508    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8569578f86-n87h2" event={"ID":"9b75006f-f808-4e0e-b478-e06c266b3322","Type":"ContainerStarted","Data":"7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf"}
May 26 16:04:31.950548 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:31.950552    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8569578f86-n87h2" event={"ID":"9b75006f-f808-4e0e-b478-e06c266b3322","Type":"ContainerStarted","Data":"cb51e6002161f45ae14d19b4d36e2e2caada39753333d87a70eb16ca9c752823"}
May 26 16:04:31.969915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:31.969865    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-8569578f86-n87h2" podStartSLOduration=1.969852696 podStartE2EDuration="1.969852696s" podCreationTimestamp="2026-05-26 16:04:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:04:31.9694343 +0000 UTC m=+223.350756345" watchObservedRunningTime="2026-05-26 16:04:31.969852696 +0000 UTC m=+223.351174714"
May 26 16:04:35.936768 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:35.936730    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:39.388623 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:39.388589    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:39.389000 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:39.388633    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:40.832487 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:40.832452    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:40.832487 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:40.832498    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:40.837077 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:40.837055    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:40.980637 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:40.980606    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8569578f86-n87h2"
May 26 16:04:50.588236 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.588170    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7c488887cf-4p4dz" podUID="b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" containerName="console" containerID="cri-o://3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230" gracePeriod=15
May 26 16:04:50.855412 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.855390    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c488887cf-4p4dz_b190ce6f-c7c6-4f37-bd4d-cf541e6270ba/console/0.log"
May 26 16:04:50.855538 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.855460    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:50.976747 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.976708    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-service-ca\") pod \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") "
May 26 16:04:50.976931 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.976778    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2vp9\" (UniqueName: \"kubernetes.io/projected/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-kube-api-access-r2vp9\") pod \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") "
May 26 16:04:50.976931 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.976811    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-serving-cert\") pod \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") "
May 26 16:04:50.976931 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.976848    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-oauth-config\") pod \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") "
May 26 16:04:50.976931 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.976923    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-oauth-serving-cert\") pod \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") "
May 26 16:04:50.977116 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.976972    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-config\") pod \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\" (UID: \"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba\") "
May 26 16:04:50.977890 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.977858    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-config" (OuterVolumeSpecName: "console-config") pod "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" (UID: "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:50.979350 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.979322    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-service-ca" (OuterVolumeSpecName: "service-ca") pod "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" (UID: "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:50.982164 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.982131    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-kube-api-access-r2vp9" (OuterVolumeSpecName: "kube-api-access-r2vp9") pod "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" (UID: "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba"). InnerVolumeSpecName "kube-api-access-r2vp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:50.984008 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.983978    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" (UID: "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:50.984429 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.984377    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" (UID: "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:50.985523 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:50.985497    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" (UID: "b190ce6f-c7c6-4f37-bd4d-cf541e6270ba"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:51.010367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.010347    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c488887cf-4p4dz_b190ce6f-c7c6-4f37-bd4d-cf541e6270ba/console/0.log"
May 26 16:04:51.010462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.010382    2570 generic.go:358] "Generic (PLEG): container finished" podID="b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" containerID="3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230" exitCode=2
May 26 16:04:51.010462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.010413    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c488887cf-4p4dz" event={"ID":"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba","Type":"ContainerDied","Data":"3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230"}
May 26 16:04:51.010462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.010435    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c488887cf-4p4dz" event={"ID":"b190ce6f-c7c6-4f37-bd4d-cf541e6270ba","Type":"ContainerDied","Data":"e70d96b260f272763d0fa4b9f7a0eea175884048cb6679602ce415db341b62da"}
May 26 16:04:51.010462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.010449    2570 scope.go:117] "RemoveContainer" containerID="3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230"
May 26 16:04:51.010617 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.010470    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c488887cf-4p4dz"
May 26 16:04:51.029516 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.029497    2570 scope.go:117] "RemoveContainer" containerID="3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230"
May 26 16:04:51.029814 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:51.029791    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230\": container with ID starting with 3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230 not found: ID does not exist" containerID="3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230"
May 26 16:04:51.029867 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.029822    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230"} err="failed to get container status \"3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230\": rpc error: code = NotFound desc = could not find container \"3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230\": container with ID starting with 3245c637b16f55ceb4c510cd49af728e0fd06ff909543a7e2279f89cfaeac230 not found: ID does not exist"
May 26 16:04:51.032370 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.032351    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c488887cf-4p4dz"]
May 26 16:04:51.035843 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.035823    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c488887cf-4p4dz"]
May 26 16:04:51.078330 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.078305    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-service-ca\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:51.078330 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.078332    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2vp9\" (UniqueName: \"kubernetes.io/projected/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-kube-api-access-r2vp9\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:51.078506 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.078346    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-serving-cert\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:51.078506 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.078359    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-oauth-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:51.078506 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.078371    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-oauth-serving-cert\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:51.078506 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.078384    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba-console-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:51.235218 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:51.235181    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" path="/var/lib/kubelet/pods/b190ce6f-c7c6-4f37-bd4d-cf541e6270ba/volumes"
May 26 16:04:55.480551 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.480509    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-9bdd488d-t42v4" podUID="d86a048e-fd12-4324-a820-ee27cf6704a3" containerName="console" containerID="cri-o://2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a" gracePeriod=15
May 26 16:04:55.750578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.750556    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9bdd488d-t42v4_d86a048e-fd12-4324-a820-ee27cf6704a3/console/0.log"
May 26 16:04:55.750679 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.750620    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:55.920727 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.920694    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-service-ca\") pod \"d86a048e-fd12-4324-a820-ee27cf6704a3\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") "
May 26 16:04:55.920900 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.920744    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-oauth-config\") pod \"d86a048e-fd12-4324-a820-ee27cf6704a3\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") "
May 26 16:04:55.920900 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.920795    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-serving-cert\") pod \"d86a048e-fd12-4324-a820-ee27cf6704a3\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") "
May 26 16:04:55.920900 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.920838    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-console-config\") pod \"d86a048e-fd12-4324-a820-ee27cf6704a3\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") "
May 26 16:04:55.920900 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.920857    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-oauth-serving-cert\") pod \"d86a048e-fd12-4324-a820-ee27cf6704a3\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") "
May 26 16:04:55.920900 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.920872    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-trusted-ca-bundle\") pod \"d86a048e-fd12-4324-a820-ee27cf6704a3\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") "
May 26 16:04:55.921139 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.920934    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5j4p\" (UniqueName: \"kubernetes.io/projected/d86a048e-fd12-4324-a820-ee27cf6704a3-kube-api-access-h5j4p\") pod \"d86a048e-fd12-4324-a820-ee27cf6704a3\" (UID: \"d86a048e-fd12-4324-a820-ee27cf6704a3\") "
May 26 16:04:55.921139 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.921102    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-service-ca" (OuterVolumeSpecName: "service-ca") pod "d86a048e-fd12-4324-a820-ee27cf6704a3" (UID: "d86a048e-fd12-4324-a820-ee27cf6704a3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:55.921358 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.921328    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d86a048e-fd12-4324-a820-ee27cf6704a3" (UID: "d86a048e-fd12-4324-a820-ee27cf6704a3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:55.921358 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.921353    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-console-config" (OuterVolumeSpecName: "console-config") pod "d86a048e-fd12-4324-a820-ee27cf6704a3" (UID: "d86a048e-fd12-4324-a820-ee27cf6704a3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:55.921561 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.921376    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d86a048e-fd12-4324-a820-ee27cf6704a3" (UID: "d86a048e-fd12-4324-a820-ee27cf6704a3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:55.921561 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.921452    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-trusted-ca-bundle\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:55.921561 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.921469    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-service-ca\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:55.921561 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.921482    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-console-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:55.921561 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.921491    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86a048e-fd12-4324-a820-ee27cf6704a3-oauth-serving-cert\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:55.923050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.923028    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d86a048e-fd12-4324-a820-ee27cf6704a3" (UID: "d86a048e-fd12-4324-a820-ee27cf6704a3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:55.923150 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.923052    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d86a048e-fd12-4324-a820-ee27cf6704a3" (UID: "d86a048e-fd12-4324-a820-ee27cf6704a3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:55.923150 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:55.923133    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86a048e-fd12-4324-a820-ee27cf6704a3-kube-api-access-h5j4p" (OuterVolumeSpecName: "kube-api-access-h5j4p") pod "d86a048e-fd12-4324-a820-ee27cf6704a3" (UID: "d86a048e-fd12-4324-a820-ee27cf6704a3"). InnerVolumeSpecName "kube-api-access-h5j4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:56.022033 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.021953    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5j4p\" (UniqueName: \"kubernetes.io/projected/d86a048e-fd12-4324-a820-ee27cf6704a3-kube-api-access-h5j4p\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:56.022033 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.021983    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-oauth-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:56.022033 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.021996    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86a048e-fd12-4324-a820-ee27cf6704a3-console-serving-cert\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:04:56.025101 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.025081    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9bdd488d-t42v4_d86a048e-fd12-4324-a820-ee27cf6704a3/console/0.log"
May 26 16:04:56.025218 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.025128    2570 generic.go:358] "Generic (PLEG): container finished" podID="d86a048e-fd12-4324-a820-ee27cf6704a3" containerID="2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a" exitCode=2
May 26 16:04:56.025218 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.025159    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bdd488d-t42v4" event={"ID":"d86a048e-fd12-4324-a820-ee27cf6704a3","Type":"ContainerDied","Data":"2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a"}
May 26 16:04:56.025218 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.025199    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bdd488d-t42v4"
May 26 16:04:56.025218 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.025214    2570 scope.go:117] "RemoveContainer" containerID="2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a"
May 26 16:04:56.025391 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.025203    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bdd488d-t42v4" event={"ID":"d86a048e-fd12-4324-a820-ee27cf6704a3","Type":"ContainerDied","Data":"9b281d8050fa301a6d5b377c70edde0efd95d5e931d2ce95c3002f095d93349d"}
May 26 16:04:56.035313 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.035294    2570 scope.go:117] "RemoveContainer" containerID="2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a"
May 26 16:04:56.035574 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:04:56.035553    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a\": container with ID starting with 2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a not found: ID does not exist" containerID="2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a"
May 26 16:04:56.035617 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.035584    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a"} err="failed to get container status \"2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a\": rpc error: code = NotFound desc = could not find container \"2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a\": container with ID starting with 2cd38b555502fbad48da315721ae68c9af9283b813e3357143b421c1bbe6347a not found: ID does not exist"
May 26 16:04:56.047841 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.047814    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bdd488d-t42v4"]
May 26 16:04:56.049991 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:56.049971    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9bdd488d-t42v4"]
May 26 16:04:57.235518 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:57.235484    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86a048e-fd12-4324-a820-ee27cf6704a3" path="/var/lib/kubelet/pods/d86a048e-fd12-4324-a820-ee27cf6704a3/volumes"
May 26 16:04:59.393765 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:59.393737    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:59.397620 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:59.397593    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7888fb4b49-lfbg2"
May 26 16:04:59.958356 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:59.958315    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:04:59.960582 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:04:59.960563    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02034fc7-cc11-48cf-8eff-ae2384a2080e-metrics-certs\") pod \"network-metrics-daemon-r88dc\" (UID: \"02034fc7-cc11-48cf-8eff-ae2384a2080e\") " pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:05:00.034398 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:00.034354    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8ddfl\""
May 26 16:05:00.043102 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:00.043083    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r88dc"
May 26 16:05:00.175985 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:00.175962    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r88dc"]
May 26 16:05:00.178694 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:05:00.178666    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02034fc7_cc11_48cf_8eff_ae2384a2080e.slice/crio-23b202f7988a3071c4fca1cf9f04f8a1bcda0d50a6867e15d9b42e66b66ba359 WatchSource:0}: Error finding container 23b202f7988a3071c4fca1cf9f04f8a1bcda0d50a6867e15d9b42e66b66ba359: Status 404 returned error can't find the container with id 23b202f7988a3071c4fca1cf9f04f8a1bcda0d50a6867e15d9b42e66b66ba359
May 26 16:05:01.039460 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:01.039418    2570 generic.go:358] "Generic (PLEG): container finished" podID="6fe2ee2b-83d4-401a-a6e1-389caeeb7e09" containerID="cd06e39a5490314a986b6c26bfc49131ecdf370bc22c789e5d0047adfdd77143" exitCode=0
May 26 16:05:01.039905 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:01.039497    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-5nqbd" event={"ID":"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09","Type":"ContainerDied","Data":"cd06e39a5490314a986b6c26bfc49131ecdf370bc22c789e5d0047adfdd77143"}
May 26 16:05:01.039979 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:01.039901    2570 scope.go:117] "RemoveContainer" containerID="cd06e39a5490314a986b6c26bfc49131ecdf370bc22c789e5d0047adfdd77143"
May 26 16:05:01.040853 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:01.040827    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r88dc" event={"ID":"02034fc7-cc11-48cf-8eff-ae2384a2080e","Type":"ContainerStarted","Data":"23b202f7988a3071c4fca1cf9f04f8a1bcda0d50a6867e15d9b42e66b66ba359"}
May 26 16:05:01.042239 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:01.042214    2570 generic.go:358] "Generic (PLEG): container finished" podID="aa82f87f-2f4a-4cd6-98cd-c72f77130748" containerID="1b9e4e3b2a603d5bc4f54ed261fdbd423eb24923346d1bc9d4536decc97843e0" exitCode=0
May 26 16:05:01.042350 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:01.042255    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb" event={"ID":"aa82f87f-2f4a-4cd6-98cd-c72f77130748","Type":"ContainerDied","Data":"1b9e4e3b2a603d5bc4f54ed261fdbd423eb24923346d1bc9d4536decc97843e0"}
May 26 16:05:01.042608 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:01.042592    2570 scope.go:117] "RemoveContainer" containerID="1b9e4e3b2a603d5bc4f54ed261fdbd423eb24923346d1bc9d4536decc97843e0"
May 26 16:05:02.049960 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:02.049927    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r88dc" event={"ID":"02034fc7-cc11-48cf-8eff-ae2384a2080e","Type":"ContainerStarted","Data":"8d217c9a99430b5329fef1c4458ee2614c06ca837165cb22f3348cc0f3df70e8"}
May 26 16:05:02.052693 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:02.052666    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-nvhtb" event={"ID":"aa82f87f-2f4a-4cd6-98cd-c72f77130748","Type":"ContainerStarted","Data":"f60ac3d8817e6188c810689447adfae9bd5e5aba48ea63b5f59f934e0bf489d0"}
May 26 16:05:02.055908 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:02.055879    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-5nqbd" event={"ID":"6fe2ee2b-83d4-401a-a6e1-389caeeb7e09","Type":"ContainerStarted","Data":"c8e02be3ab14111ecd8c8fef268c77ca2eba1a6c30a4b480a41079f3f3dcca59"}
May 26 16:05:02.245806 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:02.245769    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qm4f5_e3f547af-2cf4-4821-ae28-e2087a814658/dns-node-resolver/0.log"
May 26 16:05:03.060690 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:03.060654    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r88dc" event={"ID":"02034fc7-cc11-48cf-8eff-ae2384a2080e","Type":"ContainerStarted","Data":"0fd62eeb3b45c18f6f230473ac5aa8939afcc00f265bd95e3dcae52a1392312a"}
May 26 16:05:03.077315 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:03.077242    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r88dc" podStartSLOduration=252.391440922 podStartE2EDuration="4m14.077225253s" podCreationTimestamp="2026-05-26 16:00:49 +0000 UTC" firstStartedPulling="2026-05-26 16:05:00.180598686 +0000 UTC m=+251.561920696" lastFinishedPulling="2026-05-26 16:05:01.866383022 +0000 UTC m=+253.247705027" observedRunningTime="2026-05-26 16:05:03.07587626 +0000 UTC m=+254.457198285" watchObservedRunningTime="2026-05-26 16:05:03.077225253 +0000 UTC m=+254.458547279"
May 26 16:05:28.721673 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:05:28.721616    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2lbjs" podUID="9ea83ed2-100d-4de2-a94d-5aed66a964a7"
May 26 16:05:29.136837 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:29.136811    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2lbjs"
May 26 16:05:32.041394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.041356    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:05:32.041394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.041399    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:05:32.043691 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.043661    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea83ed2-100d-4de2-a94d-5aed66a964a7-metrics-tls\") pod \"dns-default-2lbjs\" (UID: \"9ea83ed2-100d-4de2-a94d-5aed66a964a7\") " pod="openshift-dns/dns-default-2lbjs"
May 26 16:05:32.043807 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.043764    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a560b45b-1006-463e-8922-5b41684d6458-cert\") pod \"ingress-canary-xd8kr\" (UID: \"a560b45b-1006-463e-8922-5b41684d6458\") " pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:05:32.139929 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.139898    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lppkw\""
May 26 16:05:32.148231 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.148206    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2lbjs"
May 26 16:05:32.291918 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.291894    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2lbjs"]
May 26 16:05:32.294136 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:05:32.294101    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea83ed2_100d_4de2_a94d_5aed66a964a7.slice/crio-4b289550d8e6952ae35f6324080a4a166aed24587f177fac738598e5ba14013e WatchSource:0}: Error finding container 4b289550d8e6952ae35f6324080a4a166aed24587f177fac738598e5ba14013e: Status 404 returned error can't find the container with id 4b289550d8e6952ae35f6324080a4a166aed24587f177fac738598e5ba14013e
May 26 16:05:32.334746 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.334715    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-22s8h\""
May 26 16:05:32.342668 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.342636    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xd8kr"
May 26 16:05:32.463808 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:32.463783    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xd8kr"]
May 26 16:05:32.466158 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:05:32.466129    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda560b45b_1006_463e_8922_5b41684d6458.slice/crio-32207e08ba1cd3148c78efe0287d1561d68896e99355d2c5ce35f83c23c2a222 WatchSource:0}: Error finding container 32207e08ba1cd3148c78efe0287d1561d68896e99355d2c5ce35f83c23c2a222: Status 404 returned error can't find the container with id 32207e08ba1cd3148c78efe0287d1561d68896e99355d2c5ce35f83c23c2a222
May 26 16:05:33.130581 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.130321    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f5ccf97bd-4lkkq"]
May 26 16:05:33.131293 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.131249    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86a048e-fd12-4324-a820-ee27cf6704a3" containerName="console"
May 26 16:05:33.131430 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.131301    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86a048e-fd12-4324-a820-ee27cf6704a3" containerName="console"
May 26 16:05:33.131430 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.131333    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" containerName="console"
May 26 16:05:33.131430 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.131342    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" containerName="console"
May 26 16:05:33.131595 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.131448    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="d86a048e-fd12-4324-a820-ee27cf6704a3" containerName="console"
May 26 16:05:33.131595 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.131465    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="b190ce6f-c7c6-4f37-bd4d-cf541e6270ba" containerName="console"
May 26 16:05:33.133693 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.133672    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.144563 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.144537    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5ccf97bd-4lkkq"]
May 26 16:05:33.151192 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.151146    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xd8kr" event={"ID":"a560b45b-1006-463e-8922-5b41684d6458","Type":"ContainerStarted","Data":"32207e08ba1cd3148c78efe0287d1561d68896e99355d2c5ce35f83c23c2a222"}
May 26 16:05:33.154054 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.154030    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2lbjs" event={"ID":"9ea83ed2-100d-4de2-a94d-5aed66a964a7","Type":"ContainerStarted","Data":"4b289550d8e6952ae35f6324080a4a166aed24587f177fac738598e5ba14013e"}
May 26 16:05:33.252691 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.252640    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-config\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.252691 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.252687    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67xrt\" (UniqueName: \"kubernetes.io/projected/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-kube-api-access-67xrt\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.252934 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.252762    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-serving-cert\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.252934 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.252789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-trusted-ca-bundle\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.252934 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.252817    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-oauth-serving-cert\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.252934 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.252917    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-service-ca\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.253125 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.252953    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-oauth-config\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.354129 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.354039    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-config\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.354129 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.354092    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-67xrt\" (UniqueName: \"kubernetes.io/projected/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-kube-api-access-67xrt\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.354402 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.354148    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-serving-cert\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.354402 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.354174    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-trusted-ca-bundle\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.354402 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.354200    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-oauth-serving-cert\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.354402 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.354248    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-service-ca\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.354402 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.354273    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-oauth-config\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.355465 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.354872    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-config\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.355465 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.355414    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-oauth-serving-cert\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.355465 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.355435    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-trusted-ca-bundle\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.355740 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.355712    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-service-ca\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.357467 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.357445    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-oauth-config\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.358549 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.358526    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-serving-cert\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.364271 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.364248    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xrt\" (UniqueName: \"kubernetes.io/projected/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-kube-api-access-67xrt\") pod \"console-5f5ccf97bd-4lkkq\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") " pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:33.456697 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:33.456656    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:34.411207 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:34.411180    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5ccf97bd-4lkkq"]
May 26 16:05:34.414688 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:05:34.414429    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6812c0ca_d67e_4bab_b0d8_e3944eb1fbe8.slice/crio-792b09a5b157e0e9da1cc2eac35d9c9602accba3620e80e69e57015b4cb26487 WatchSource:0}: Error finding container 792b09a5b157e0e9da1cc2eac35d9c9602accba3620e80e69e57015b4cb26487: Status 404 returned error can't find the container with id 792b09a5b157e0e9da1cc2eac35d9c9602accba3620e80e69e57015b4cb26487
May 26 16:05:35.045901 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.045865    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:05:35.046314 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.046290    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="alertmanager" containerID="cri-o://567894c1b53c77cdb146e7d502e9b90cdeb35970a5575dd4a71a8c26818c1274" gracePeriod=120
May 26 16:05:35.046436 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.046360    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy-web" containerID="cri-o://919ff67912e4c7f265c2ade9ebdadd33bb576980a6446b77c1a388a0deb4e97d" gracePeriod=120
May 26 16:05:35.046436 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.046359    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy-metric" containerID="cri-o://551232802f169c3f4965f042d052ba2c540c85d68136b087175a202611b08a5c" gracePeriod=120
May 26 16:05:35.046436 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.046387    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy" containerID="cri-o://8c8a673d0910dd4a859cc51c8c10a394ffb8d9417df12d25cf615a20337634ff" gracePeriod=120
May 26 16:05:35.046612 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.046439    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="prom-label-proxy" containerID="cri-o://9758421cc20ac998332d48ff7a2d42217cdae0cf306e7a627d9e234c04ce723f" gracePeriod=120
May 26 16:05:35.046612 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.046428    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="config-reloader" containerID="cri-o://2e0d0d5bce3ae419dcb0e353c28475f881fe52e141b536575108910f00432832" gracePeriod=120
May 26 16:05:35.164953 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.164927    2570 generic.go:358] "Generic (PLEG): container finished" podID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerID="9758421cc20ac998332d48ff7a2d42217cdae0cf306e7a627d9e234c04ce723f" exitCode=0
May 26 16:05:35.164953 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.164952    2570 generic.go:358] "Generic (PLEG): container finished" podID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerID="551232802f169c3f4965f042d052ba2c540c85d68136b087175a202611b08a5c" exitCode=0
May 26 16:05:35.165114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.164961    2570 generic.go:358] "Generic (PLEG): container finished" podID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerID="8c8a673d0910dd4a859cc51c8c10a394ffb8d9417df12d25cf615a20337634ff" exitCode=0
May 26 16:05:35.165114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.164970    2570 generic.go:358] "Generic (PLEG): container finished" podID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerID="2e0d0d5bce3ae419dcb0e353c28475f881fe52e141b536575108910f00432832" exitCode=0
May 26 16:05:35.165114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.164978    2570 generic.go:358] "Generic (PLEG): container finished" podID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerID="567894c1b53c77cdb146e7d502e9b90cdeb35970a5575dd4a71a8c26818c1274" exitCode=0
May 26 16:05:35.165114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.164994    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerDied","Data":"9758421cc20ac998332d48ff7a2d42217cdae0cf306e7a627d9e234c04ce723f"}
May 26 16:05:35.165114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.165029    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerDied","Data":"551232802f169c3f4965f042d052ba2c540c85d68136b087175a202611b08a5c"}
May 26 16:05:35.165114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.165040    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerDied","Data":"8c8a673d0910dd4a859cc51c8c10a394ffb8d9417df12d25cf615a20337634ff"}
May 26 16:05:35.165114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.165048    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerDied","Data":"2e0d0d5bce3ae419dcb0e353c28475f881fe52e141b536575108910f00432832"}
May 26 16:05:35.165114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.165057    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerDied","Data":"567894c1b53c77cdb146e7d502e9b90cdeb35970a5575dd4a71a8c26818c1274"}
May 26 16:05:35.166518 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.166498    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2lbjs" event={"ID":"9ea83ed2-100d-4de2-a94d-5aed66a964a7","Type":"ContainerStarted","Data":"71d644a5bf338aefc00e4cf65c16610e8bf97b6ba6d14f5ec6e97be475f7aefe"}
May 26 16:05:35.166590 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.166524    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2lbjs" event={"ID":"9ea83ed2-100d-4de2-a94d-5aed66a964a7","Type":"ContainerStarted","Data":"8d13c3ab0ff789b58ee707d21c02e4f33392d253ac753b4cd1cc77a67df4ba19"}
May 26 16:05:35.166590 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.166581    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2lbjs"
May 26 16:05:35.167786 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.167764    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5ccf97bd-4lkkq" event={"ID":"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8","Type":"ContainerStarted","Data":"e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e"}
May 26 16:05:35.167860 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.167789    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5ccf97bd-4lkkq" event={"ID":"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8","Type":"ContainerStarted","Data":"792b09a5b157e0e9da1cc2eac35d9c9602accba3620e80e69e57015b4cb26487"}
May 26 16:05:35.168984 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.168965    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xd8kr" event={"ID":"a560b45b-1006-463e-8922-5b41684d6458","Type":"ContainerStarted","Data":"f5ae56e6fec10aa5d9347fb71e717be22f3f640363e19a273abd613c2eb777a8"}
May 26 16:05:35.183198 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.183159    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-2lbjs" podStartSLOduration=252.222743124 podStartE2EDuration="4m14.183148092s" podCreationTimestamp="2026-05-26 16:01:21 +0000 UTC" firstStartedPulling="2026-05-26 16:05:32.295962958 +0000 UTC m=+283.677284963" lastFinishedPulling="2026-05-26 16:05:34.256367913 +0000 UTC m=+285.637689931" observedRunningTime="2026-05-26 16:05:35.181312041 +0000 UTC m=+286.562634065" watchObservedRunningTime="2026-05-26 16:05:35.183148092 +0000 UTC m=+286.564470116"
May 26 16:05:35.197415 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.197377    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5f5ccf97bd-4lkkq" podStartSLOduration=2.197363186 podStartE2EDuration="2.197363186s" podCreationTimestamp="2026-05-26 16:05:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:05:35.195934541 +0000 UTC m=+286.577256567" watchObservedRunningTime="2026-05-26 16:05:35.197363186 +0000 UTC m=+286.578685212"
May 26 16:05:35.210773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:35.210721    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xd8kr" podStartSLOduration=252.418102936 podStartE2EDuration="4m14.210706321s" podCreationTimestamp="2026-05-26 16:01:21 +0000 UTC" firstStartedPulling="2026-05-26 16:05:32.468028563 +0000 UTC m=+283.849350566" lastFinishedPulling="2026-05-26 16:05:34.260631939 +0000 UTC m=+285.641953951" observedRunningTime="2026-05-26 16:05:35.210023712 +0000 UTC m=+286.591345740" watchObservedRunningTime="2026-05-26 16:05:35.210706321 +0000 UTC m=+286.592028348"
May 26 16:05:36.175993 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.175961    2570 generic.go:358] "Generic (PLEG): container finished" podID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerID="919ff67912e4c7f265c2ade9ebdadd33bb576980a6446b77c1a388a0deb4e97d" exitCode=0
May 26 16:05:36.176370 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.176005    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerDied","Data":"919ff67912e4c7f265c2ade9ebdadd33bb576980a6446b77c1a388a0deb4e97d"}
May 26 16:05:36.291785 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.291765    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:36.382330 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382234    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-out\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382330 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382271    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-metrics-client-ca\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382330 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382316    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-main-tls\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382588 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382335    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-tls-assets\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382588 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382361    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-285xd\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-kube-api-access-285xd\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382588 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382386    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-main-db\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382588 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382467    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-cluster-tls-config\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382588 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382536    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-trusted-ca-bundle\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382588 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382568    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382881 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382616    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-web-config\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382881 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382697    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-volume\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382881 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382724    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-web\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382881 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382724    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:05:36.382881 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382752    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\" (UID: \"146de6bf-dc8f-46fa-bd57-1f1647d543fa\") "
May 26 16:05:36.382881 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382852    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:36.383161 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.382918    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:36.383161 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.383078    2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-main-db\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.383161 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.383102    2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.383161 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.383126    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146de6bf-dc8f-46fa-bd57-1f1647d543fa-metrics-client-ca\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.385196 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.385158    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-kube-api-access-285xd" (OuterVolumeSpecName: "kube-api-access-285xd") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "kube-api-access-285xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:36.385401 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.385368    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:36.385778 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.385733    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:36.385930 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.385892    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:36.386008 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.385988    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-out" (OuterVolumeSpecName: "config-out") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:05:36.386208 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.386181    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:36.386440 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.386421    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:36.386818 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.386802    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-volume" (OuterVolumeSpecName: "config-volume") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:36.390843 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.390821    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:36.398382 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.398142    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-web-config" (OuterVolumeSpecName: "web-config") pod "146de6bf-dc8f-46fa-bd57-1f1647d543fa" (UID: "146de6bf-dc8f-46fa-bd57-1f1647d543fa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:36.484290 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484253    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.484425 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484303    2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-web-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.484425 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484315    2570 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-volume\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.484425 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484324    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.484425 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484333    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.484425 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484342    2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/146de6bf-dc8f-46fa-bd57-1f1647d543fa-config-out\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.484425 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484352    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-secret-alertmanager-main-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.484425 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484360    2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-tls-assets\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.484425 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484369    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-285xd\" (UniqueName: \"kubernetes.io/projected/146de6bf-dc8f-46fa-bd57-1f1647d543fa-kube-api-access-285xd\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:36.484425 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:36.484377    2570 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146de6bf-dc8f-46fa-bd57-1f1647d543fa-cluster-tls-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:05:37.182116 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.182082    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"146de6bf-dc8f-46fa-bd57-1f1647d543fa","Type":"ContainerDied","Data":"f989060d8d208c1ad3abdf7b5c362ef74d5332bc0dbbf857b1083c9b08269608"}
May 26 16:05:37.182590 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.182130    2570 scope.go:117] "RemoveContainer" containerID="9758421cc20ac998332d48ff7a2d42217cdae0cf306e7a627d9e234c04ce723f"
May 26 16:05:37.182590 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.182193    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.190271 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.190045    2570 scope.go:117] "RemoveContainer" containerID="551232802f169c3f4965f042d052ba2c540c85d68136b087175a202611b08a5c"
May 26 16:05:37.196800 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.196785    2570 scope.go:117] "RemoveContainer" containerID="8c8a673d0910dd4a859cc51c8c10a394ffb8d9417df12d25cf615a20337634ff"
May 26 16:05:37.205058 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.205032    2570 scope.go:117] "RemoveContainer" containerID="919ff67912e4c7f265c2ade9ebdadd33bb576980a6446b77c1a388a0deb4e97d"
May 26 16:05:37.206420 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.206399    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:05:37.209882 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.209862    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:05:37.212389 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.212376    2570 scope.go:117] "RemoveContainer" containerID="2e0d0d5bce3ae419dcb0e353c28475f881fe52e141b536575108910f00432832"
May 26 16:05:37.218437 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.218424    2570 scope.go:117] "RemoveContainer" containerID="567894c1b53c77cdb146e7d502e9b90cdeb35970a5575dd4a71a8c26818c1274"
May 26 16:05:37.224778 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.224738    2570 scope.go:117] "RemoveContainer" containerID="511adab247fd48261fbd24cb45a7dd804f8ca70fa447ebf0a2dc12e01ff276e4"
May 26 16:05:37.235327 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.235303    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" path="/var/lib/kubelet/pods/146de6bf-dc8f-46fa-bd57-1f1647d543fa/volumes"
May 26 16:05:37.240805 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.240784    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:05:37.241119 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241104    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="alertmanager"
May 26 16:05:37.241205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241122    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="alertmanager"
May 26 16:05:37.241205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241136    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy-web"
May 26 16:05:37.241205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241146    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy-web"
May 26 16:05:37.241205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241156    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy"
May 26 16:05:37.241205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241163    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy"
May 26 16:05:37.241205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241172    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy-metric"
May 26 16:05:37.241205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241181    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy-metric"
May 26 16:05:37.241205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241199    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="config-reloader"
May 26 16:05:37.241205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241207    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="config-reloader"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241225    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="init-config-reloader"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241234    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="init-config-reloader"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241244    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="prom-label-proxy"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241251    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="prom-label-proxy"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241351    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy-metric"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241365    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="alertmanager"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241375    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy-web"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241385    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="config-reloader"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241394    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="prom-label-proxy"
May 26 16:05:37.241633 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.241407    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="146de6bf-dc8f-46fa-bd57-1f1647d543fa" containerName="kube-rbac-proxy"
May 26 16:05:37.244541 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.244523    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.246788 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.246771    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 26 16:05:37.246945 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.246927    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xbwld\""
May 26 16:05:37.247109 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.247092    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 26 16:05:37.247231 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.247215    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 26 16:05:37.247303 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.247215    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 26 16:05:37.247303 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.247238    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 26 16:05:37.247617 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.247600    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 26 16:05:37.247917 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.247896    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 26 16:05:37.248369 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.248352    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 26 16:05:37.252033 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.252011    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 26 16:05:37.255808 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.255786    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:05:37.391409 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391371    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391570 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391438    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn7pd\" (UniqueName: \"kubernetes.io/projected/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-kube-api-access-pn7pd\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391570 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391479    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391570 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391533    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-config-out\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391676 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391571    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-config-volume\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391676 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391590    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391676 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391609    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391676 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391626    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391676 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391642    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391845 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391683    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391845 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391724    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391845 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391743    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-web-config\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.391845 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.391759    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492328 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492231    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7pd\" (UniqueName: \"kubernetes.io/projected/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-kube-api-access-pn7pd\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492328 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492305    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492523 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492457    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-config-out\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492523 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492493    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-config-volume\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492523 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492510    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492666 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492527    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492666 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492545    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492666 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492566    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492666 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492601    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492666 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492662    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492907 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492691    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-web-config\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492907 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492719    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.492907 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.492783    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.493187 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.493156    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.493410 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.493386    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.493674 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.493649    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.495534 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.495419    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-config-out\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.495690 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.495667    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.495789 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.495762    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.495864 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.495792    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-config-volume\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.495864 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.495792    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-web-config\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.495979 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.495949    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.496198 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.496180    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.496372 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.496354    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.497466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.497449    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.500002 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.499983    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn7pd\" (UniqueName: \"kubernetes.io/projected/d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8-kube-api-access-pn7pd\") pod \"alertmanager-main-0\" (UID: \"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.555221 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.555184    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:05:37.686176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:37.686151    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:05:37.687468 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:05:37.687441    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63d95a3_fa42_4bc0_bb47_f7453a9e4ec8.slice/crio-1913b076af9e65f091b893f7bda3ca695e072f119b26a29662c4f4f1725e58e1 WatchSource:0}: Error finding container 1913b076af9e65f091b893f7bda3ca695e072f119b26a29662c4f4f1725e58e1: Status 404 returned error can't find the container with id 1913b076af9e65f091b893f7bda3ca695e072f119b26a29662c4f4f1725e58e1
May 26 16:05:38.188181 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:38.188148    2570 generic.go:358] "Generic (PLEG): container finished" podID="d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8" containerID="1cfb8f97057232af7ad9eb8c2428a68be585d992110ae22c634f5f029fb0c380" exitCode=0
May 26 16:05:38.188578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:38.188231    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8","Type":"ContainerDied","Data":"1cfb8f97057232af7ad9eb8c2428a68be585d992110ae22c634f5f029fb0c380"}
May 26 16:05:38.188578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:38.188269    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8","Type":"ContainerStarted","Data":"1913b076af9e65f091b893f7bda3ca695e072f119b26a29662c4f4f1725e58e1"}
May 26 16:05:39.195635 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:39.195602    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8","Type":"ContainerStarted","Data":"77b7816c2d4b7f7118911134abc8df31c7be1ef8b702f2d3e9cf1ad5e1d3165c"}
May 26 16:05:39.195635 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:39.195636    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8","Type":"ContainerStarted","Data":"c866a9e68863a8d7c7298264d8de797c1e19a9820626b44ab669b4b07a631f9d"}
May 26 16:05:39.196056 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:39.195649    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8","Type":"ContainerStarted","Data":"3c42e531390630921a710bcda7190b938a6ef37a4646015fcf60173006ae9d0c"}
May 26 16:05:39.196056 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:39.195659    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8","Type":"ContainerStarted","Data":"581a5f25f4824087fd3bac74acecd7a044bf3c296894e22d70768e94ca68697d"}
May 26 16:05:39.196056 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:39.195667    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8","Type":"ContainerStarted","Data":"291d8f4099568e63047bb0d2b842221fd172a8d1400fbfbdd5740612bde8f18e"}
May 26 16:05:39.196056 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:39.195675    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8","Type":"ContainerStarted","Data":"5e503c77afce995eed24fba37f4d520d37c6b25b9b8b5e95226ea8e6919a57f8"}
May 26 16:05:39.220763 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:39.220720    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.220707484 podStartE2EDuration="2.220707484s" podCreationTimestamp="2026-05-26 16:05:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:05:39.220089186 +0000 UTC m=+290.601411211" watchObservedRunningTime="2026-05-26 16:05:39.220707484 +0000 UTC m=+290.602029509"
May 26 16:05:43.457803 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:43.457766    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:43.457803 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:43.457809    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:43.462510 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:43.462487    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:44.221203 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:44.221171    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:05:44.267481 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:44.267450    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8569578f86-n87h2"]
May 26 16:05:45.178874 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:45.178842    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2lbjs"
May 26 16:05:49.160667 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:05:49.160648    2570 kubelet.go:1628] "Image garbage collection succeeded"
May 26 16:06:09.288967 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.288928    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-8569578f86-n87h2" podUID="9b75006f-f808-4e0e-b478-e06c266b3322" containerName="console" containerID="cri-o://7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf" gracePeriod=15
May 26 16:06:09.529138 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.529111    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8569578f86-n87h2_9b75006f-f808-4e0e-b478-e06c266b3322/console/0.log"
May 26 16:06:09.529256 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.529180    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8569578f86-n87h2"
May 26 16:06:09.656327 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656222    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-console-config\") pod \"9b75006f-f808-4e0e-b478-e06c266b3322\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") "
May 26 16:06:09.656508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656357    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-oauth-serving-cert\") pod \"9b75006f-f808-4e0e-b478-e06c266b3322\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") "
May 26 16:06:09.656508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656390    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-oauth-config\") pod \"9b75006f-f808-4e0e-b478-e06c266b3322\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") "
May 26 16:06:09.656508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656438    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-trusted-ca-bundle\") pod \"9b75006f-f808-4e0e-b478-e06c266b3322\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") "
May 26 16:06:09.656508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656467    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-serving-cert\") pod \"9b75006f-f808-4e0e-b478-e06c266b3322\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") "
May 26 16:06:09.656508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656487    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn5px\" (UniqueName: \"kubernetes.io/projected/9b75006f-f808-4e0e-b478-e06c266b3322-kube-api-access-fn5px\") pod \"9b75006f-f808-4e0e-b478-e06c266b3322\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") "
May 26 16:06:09.656508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656504    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-service-ca\") pod \"9b75006f-f808-4e0e-b478-e06c266b3322\" (UID: \"9b75006f-f808-4e0e-b478-e06c266b3322\") "
May 26 16:06:09.656788 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656550    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-console-config" (OuterVolumeSpecName: "console-config") pod "9b75006f-f808-4e0e-b478-e06c266b3322" (UID: "9b75006f-f808-4e0e-b478-e06c266b3322"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:06:09.656788 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656758    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-console-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:06:09.656788 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656765    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b75006f-f808-4e0e-b478-e06c266b3322" (UID: "9b75006f-f808-4e0e-b478-e06c266b3322"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:06:09.656987 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.656863    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9b75006f-f808-4e0e-b478-e06c266b3322" (UID: "9b75006f-f808-4e0e-b478-e06c266b3322"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:06:09.657048 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.657021    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b75006f-f808-4e0e-b478-e06c266b3322" (UID: "9b75006f-f808-4e0e-b478-e06c266b3322"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:06:09.658636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.658614    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b75006f-f808-4e0e-b478-e06c266b3322" (UID: "9b75006f-f808-4e0e-b478-e06c266b3322"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:06:09.659002 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.658976    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b75006f-f808-4e0e-b478-e06c266b3322-kube-api-access-fn5px" (OuterVolumeSpecName: "kube-api-access-fn5px") pod "9b75006f-f808-4e0e-b478-e06c266b3322" (UID: "9b75006f-f808-4e0e-b478-e06c266b3322"). InnerVolumeSpecName "kube-api-access-fn5px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:06:09.659002 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.658990    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b75006f-f808-4e0e-b478-e06c266b3322" (UID: "9b75006f-f808-4e0e-b478-e06c266b3322"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:06:09.757191 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.757160    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-oauth-serving-cert\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:06:09.757191 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.757190    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-oauth-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:06:09.757410 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.757201    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-trusted-ca-bundle\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:06:09.757410 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.757211    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b75006f-f808-4e0e-b478-e06c266b3322-console-serving-cert\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:06:09.757410 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.757220    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5px\" (UniqueName: \"kubernetes.io/projected/9b75006f-f808-4e0e-b478-e06c266b3322-kube-api-access-fn5px\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:06:09.757410 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:09.757231    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b75006f-f808-4e0e-b478-e06c266b3322-service-ca\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:06:10.298176 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.298146    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8569578f86-n87h2_9b75006f-f808-4e0e-b478-e06c266b3322/console/0.log"
May 26 16:06:10.298608 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.298192    2570 generic.go:358] "Generic (PLEG): container finished" podID="9b75006f-f808-4e0e-b478-e06c266b3322" containerID="7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf" exitCode=2
May 26 16:06:10.298608 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.298292    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8569578f86-n87h2" event={"ID":"9b75006f-f808-4e0e-b478-e06c266b3322","Type":"ContainerDied","Data":"7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf"}
May 26 16:06:10.298608 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.298329    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8569578f86-n87h2" event={"ID":"9b75006f-f808-4e0e-b478-e06c266b3322","Type":"ContainerDied","Data":"cb51e6002161f45ae14d19b4d36e2e2caada39753333d87a70eb16ca9c752823"}
May 26 16:06:10.298608 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.298344    2570 scope.go:117] "RemoveContainer" containerID="7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf"
May 26 16:06:10.298608 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.298298    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8569578f86-n87h2"
May 26 16:06:10.306800 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.306780    2570 scope.go:117] "RemoveContainer" containerID="7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf"
May 26 16:06:10.307031 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:06:10.307014    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf\": container with ID starting with 7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf not found: ID does not exist" containerID="7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf"
May 26 16:06:10.307093 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.307044    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf"} err="failed to get container status \"7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf\": rpc error: code = NotFound desc = could not find container \"7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf\": container with ID starting with 7037272f94d9135e7622fc88da953086c3b7f6faa6920aabf7a4a354e97b5caf not found: ID does not exist"
May 26 16:06:10.319465 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.319443    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8569578f86-n87h2"]
May 26 16:06:10.323413 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:10.323394    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8569578f86-n87h2"]
May 26 16:06:11.235812 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:06:11.235780    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b75006f-f808-4e0e-b478-e06c266b3322" path="/var/lib/kubelet/pods/9b75006f-f808-4e0e-b478-e06c266b3322/volumes"
May 26 16:09:49.109062 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.108979    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7586f45f75-hth66"]
May 26 16:09:49.109572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.109358    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b75006f-f808-4e0e-b478-e06c266b3322" containerName="console"
May 26 16:09:49.109572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.109372    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b75006f-f808-4e0e-b478-e06c266b3322" containerName="console"
May 26 16:09:49.109572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.109431    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="9b75006f-f808-4e0e-b478-e06c266b3322" containerName="console"
May 26 16:09:49.112085 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.112064    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.128518 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.128494    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7586f45f75-hth66"]
May 26 16:09:49.177327 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.177299    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-trusted-ca-bundle\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.177490 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.177338    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-oauth-serving-cert\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.177490 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.177367    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53e8a5bc-366b-4b32-911c-0adaaa222933-console-oauth-config\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.177490 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.177401    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-console-config\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.177490 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.177476    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgr5\" (UniqueName: \"kubernetes.io/projected/53e8a5bc-366b-4b32-911c-0adaaa222933-kube-api-access-rkgr5\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.177625 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.177550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53e8a5bc-366b-4b32-911c-0adaaa222933-console-serving-cert\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.177625 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.177603    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-service-ca\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.278050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.278018    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53e8a5bc-366b-4b32-911c-0adaaa222933-console-oauth-config\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.278249 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.278072    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-console-config\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.278249 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.278111    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgr5\" (UniqueName: \"kubernetes.io/projected/53e8a5bc-366b-4b32-911c-0adaaa222933-kube-api-access-rkgr5\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.278249 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.278134    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53e8a5bc-366b-4b32-911c-0adaaa222933-console-serving-cert\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.278459 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.278308    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-service-ca\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.278459 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.278359    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-trusted-ca-bundle\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.278459 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.278389    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-oauth-serving-cert\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.279106 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.279080    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-console-config\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.279265 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.279119    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-oauth-serving-cert\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.279265 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.279226    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-service-ca\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.279492 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.279467    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53e8a5bc-366b-4b32-911c-0adaaa222933-trusted-ca-bundle\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.280992 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.280973    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53e8a5bc-366b-4b32-911c-0adaaa222933-console-serving-cert\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.281100 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.281005    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53e8a5bc-366b-4b32-911c-0adaaa222933-console-oauth-config\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.286805 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.286786    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgr5\" (UniqueName: \"kubernetes.io/projected/53e8a5bc-366b-4b32-911c-0adaaa222933-kube-api-access-rkgr5\") pod \"console-7586f45f75-hth66\" (UID: \"53e8a5bc-366b-4b32-911c-0adaaa222933\") " pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.421209 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.421167    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:49.546520 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.546462    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7586f45f75-hth66"]
May 26 16:09:49.548728 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:09:49.548693    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e8a5bc_366b_4b32_911c_0adaaa222933.slice/crio-766266d94abd9284fef6a465e1c6e0e91c77b4df7244a7c299c47f076c200d81 WatchSource:0}: Error finding container 766266d94abd9284fef6a465e1c6e0e91c77b4df7244a7c299c47f076c200d81: Status 404 returned error can't find the container with id 766266d94abd9284fef6a465e1c6e0e91c77b4df7244a7c299c47f076c200d81
May 26 16:09:49.550675 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.550656    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:09:49.920533 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.920494    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7586f45f75-hth66" event={"ID":"53e8a5bc-366b-4b32-911c-0adaaa222933","Type":"ContainerStarted","Data":"346df190207f73aaabd875b5602b2a48857ff9f4c11b430d3dd046846b9ce831"}
May 26 16:09:49.920707 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.920541    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7586f45f75-hth66" event={"ID":"53e8a5bc-366b-4b32-911c-0adaaa222933","Type":"ContainerStarted","Data":"766266d94abd9284fef6a465e1c6e0e91c77b4df7244a7c299c47f076c200d81"}
May 26 16:09:49.938329 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:49.938267    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7586f45f75-hth66" podStartSLOduration=0.938251374 podStartE2EDuration="938.251374ms" podCreationTimestamp="2026-05-26 16:09:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:09:49.936871002 +0000 UTC m=+541.318193026" watchObservedRunningTime="2026-05-26 16:09:49.938251374 +0000 UTC m=+541.319573400"
May 26 16:09:59.421952 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:59.421912    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:59.422454 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:59.421979    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:59.426754 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:59.426726    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7586f45f75-hth66"
May 26 16:09:59.958218 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:09:59.958190    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7586f45f75-hth66"
May 26 16:10:00.003788 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:00.003754    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f5ccf97bd-4lkkq"]
May 26 16:10:25.024883 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.024841    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5f5ccf97bd-4lkkq" podUID="6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" containerName="console" containerID="cri-o://e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e" gracePeriod=15
May 26 16:10:25.263645 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.263624    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f5ccf97bd-4lkkq_6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8/console/0.log"
May 26 16:10:25.263756 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.263686    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:10:25.292462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.292397    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-oauth-config\") pod \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") "
May 26 16:10:25.292462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.292436    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67xrt\" (UniqueName: \"kubernetes.io/projected/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-kube-api-access-67xrt\") pod \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") "
May 26 16:10:25.292462 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.292462    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-trusted-ca-bundle\") pod \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") "
May 26 16:10:25.292711 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.292567    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-oauth-serving-cert\") pod \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") "
May 26 16:10:25.292711 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.292606    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-config\") pod \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") "
May 26 16:10:25.292711 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.292682    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-service-ca\") pod \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") "
May 26 16:10:25.292859 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.292751    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-serving-cert\") pod \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\" (UID: \"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8\") "
May 26 16:10:25.292859 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.292824    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" (UID: "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:10:25.292978 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.292953    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" (UID: "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:10:25.293150 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.293120    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-config" (OuterVolumeSpecName: "console-config") pod "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" (UID: "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:10:25.293266 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.293161    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-service-ca" (OuterVolumeSpecName: "service-ca") pod "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" (UID: "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:10:25.293370 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.293307    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-service-ca\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:10:25.293370 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.293329    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-trusted-ca-bundle\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:10:25.293370 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.293351    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-oauth-serving-cert\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:10:25.293370 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.293368    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:10:25.294820 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.294793    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" (UID: "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:10:25.294923 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.294885    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" (UID: "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:10:25.295060 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.295044    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-kube-api-access-67xrt" (OuterVolumeSpecName: "kube-api-access-67xrt") pod "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" (UID: "6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8"). InnerVolumeSpecName "kube-api-access-67xrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:10:25.394254 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.394225    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67xrt\" (UniqueName: \"kubernetes.io/projected/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-kube-api-access-67xrt\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:10:25.394254 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.394251    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-serving-cert\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:10:25.394254 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:25.394261    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8-console-oauth-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:10:26.029496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.029468    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f5ccf97bd-4lkkq_6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8/console/0.log"
May 26 16:10:26.029924 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.029510    2570 generic.go:358] "Generic (PLEG): container finished" podID="6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" containerID="e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e" exitCode=2
May 26 16:10:26.029924 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.029571    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5ccf97bd-4lkkq" event={"ID":"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8","Type":"ContainerDied","Data":"e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e"}
May 26 16:10:26.029924 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.029585    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5ccf97bd-4lkkq"
May 26 16:10:26.029924 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.029604    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5ccf97bd-4lkkq" event={"ID":"6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8","Type":"ContainerDied","Data":"792b09a5b157e0e9da1cc2eac35d9c9602accba3620e80e69e57015b4cb26487"}
May 26 16:10:26.029924 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.029624    2570 scope.go:117] "RemoveContainer" containerID="e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e"
May 26 16:10:26.038009 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.037989    2570 scope.go:117] "RemoveContainer" containerID="e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e"
May 26 16:10:26.038247 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:10:26.038229    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e\": container with ID starting with e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e not found: ID does not exist" containerID="e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e"
May 26 16:10:26.038338 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.038256    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e"} err="failed to get container status \"e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e\": rpc error: code = NotFound desc = could not find container \"e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e\": container with ID starting with e80617bab8437a3233201f75ab557e5efeb3085bd89c0fe950060a8b93c7bb1e not found: ID does not exist"
May 26 16:10:26.051559 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.051532    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f5ccf97bd-4lkkq"]
May 26 16:10:26.057209 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:26.057187    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f5ccf97bd-4lkkq"]
May 26 16:10:27.235026 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:10:27.234991    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" path="/var/lib/kubelet/pods/6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8/volumes"
May 26 16:11:23.922005 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.921965    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"]
May 26 16:11:23.924499 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.922432    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" containerName="console"
May 26 16:11:23.924499 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.922451    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" containerName="console"
May 26 16:11:23.924499 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.922512    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="6812c0ca-d67e-4bab-b0d8-e3944eb1fbe8" containerName="console"
May 26 16:11:23.925434 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.925417    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:23.927693 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.927673    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 26 16:11:23.927797 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.927730    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 26 16:11:23.928403 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.928389    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qk2s9\""
May 26 16:11:23.933100 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.933076    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"]
May 26 16:11:23.983843 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.983805    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:23.983995 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.983878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:23.983995 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:23.983938    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwtm\" (UniqueName: \"kubernetes.io/projected/0da4fcc8-2459-40c3-bab7-74ce418d83b5-kube-api-access-5hwtm\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:24.084944 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:24.084910    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:24.085061 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:24.084958    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwtm\" (UniqueName: \"kubernetes.io/projected/0da4fcc8-2459-40c3-bab7-74ce418d83b5-kube-api-access-5hwtm\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:24.085061 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:24.084986    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:24.085298 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:24.085258    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:24.085374 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:24.085359    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:24.094136 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:24.094102    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwtm\" (UniqueName: \"kubernetes.io/projected/0da4fcc8-2459-40c3-bab7-74ce418d83b5-kube-api-access-5hwtm\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:24.234994 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:24.234918    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:24.358527 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:24.358500    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"]
May 26 16:11:24.360959 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:11:24.360929    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da4fcc8_2459_40c3_bab7_74ce418d83b5.slice/crio-3c13c6af8075d816f31a2de2fcad0ff104f56845777567a543f19157f772d7bc WatchSource:0}: Error finding container 3c13c6af8075d816f31a2de2fcad0ff104f56845777567a543f19157f772d7bc: Status 404 returned error can't find the container with id 3c13c6af8075d816f31a2de2fcad0ff104f56845777567a543f19157f772d7bc
May 26 16:11:25.207235 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:25.207192    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w" event={"ID":"0da4fcc8-2459-40c3-bab7-74ce418d83b5","Type":"ContainerStarted","Data":"3c13c6af8075d816f31a2de2fcad0ff104f56845777567a543f19157f772d7bc"}
May 26 16:11:32.232744 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:32.232704    2570 generic.go:358] "Generic (PLEG): container finished" podID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerID="ae9cad7620cba01f5d88bb9658be4fdb6ff8e0e9a4f44651058e9dc13e0fabfe" exitCode=0
May 26 16:11:32.233131 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:32.232790    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w" event={"ID":"0da4fcc8-2459-40c3-bab7-74ce418d83b5","Type":"ContainerDied","Data":"ae9cad7620cba01f5d88bb9658be4fdb6ff8e0e9a4f44651058e9dc13e0fabfe"}
May 26 16:11:35.242744 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:35.242713    2570 generic.go:358] "Generic (PLEG): container finished" podID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerID="195245936d13194d88f3c12e77b3c2ce058fff3d1b1d8567f92896f968c8c887" exitCode=0
May 26 16:11:35.243121 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:35.242780    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w" event={"ID":"0da4fcc8-2459-40c3-bab7-74ce418d83b5","Type":"ContainerDied","Data":"195245936d13194d88f3c12e77b3c2ce058fff3d1b1d8567f92896f968c8c887"}
May 26 16:11:41.263685 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:41.263647    2570 generic.go:358] "Generic (PLEG): container finished" podID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerID="04cf995fe846559287ff55522b080633930cc611330865263f35bbb43e7f662e" exitCode=0
May 26 16:11:41.264107 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:41.263729    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w" event={"ID":"0da4fcc8-2459-40c3-bab7-74ce418d83b5","Type":"ContainerDied","Data":"04cf995fe846559287ff55522b080633930cc611330865263f35bbb43e7f662e"}
May 26 16:11:42.385027 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.385004    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:42.454884 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.454851    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-util\") pod \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") "
May 26 16:11:42.455051 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.454935    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-bundle\") pod \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") "
May 26 16:11:42.455051 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.455002    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hwtm\" (UniqueName: \"kubernetes.io/projected/0da4fcc8-2459-40c3-bab7-74ce418d83b5-kube-api-access-5hwtm\") pod \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\" (UID: \"0da4fcc8-2459-40c3-bab7-74ce418d83b5\") "
May 26 16:11:42.455510 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.455490    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-bundle" (OuterVolumeSpecName: "bundle") pod "0da4fcc8-2459-40c3-bab7-74ce418d83b5" (UID: "0da4fcc8-2459-40c3-bab7-74ce418d83b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:11:42.457119 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.457101    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da4fcc8-2459-40c3-bab7-74ce418d83b5-kube-api-access-5hwtm" (OuterVolumeSpecName: "kube-api-access-5hwtm") pod "0da4fcc8-2459-40c3-bab7-74ce418d83b5" (UID: "0da4fcc8-2459-40c3-bab7-74ce418d83b5"). InnerVolumeSpecName "kube-api-access-5hwtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:11:42.460550 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.460498    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-util" (OuterVolumeSpecName: "util") pod "0da4fcc8-2459-40c3-bab7-74ce418d83b5" (UID: "0da4fcc8-2459-40c3-bab7-74ce418d83b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:11:42.555824 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.555733    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwtm\" (UniqueName: \"kubernetes.io/projected/0da4fcc8-2459-40c3-bab7-74ce418d83b5-kube-api-access-5hwtm\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:11:42.555824 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.555764    2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-util\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:11:42.555824 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:42.555779    2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0da4fcc8-2459-40c3-bab7-74ce418d83b5-bundle\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:11:43.270685 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:43.270655    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w" event={"ID":"0da4fcc8-2459-40c3-bab7-74ce418d83b5","Type":"ContainerDied","Data":"3c13c6af8075d816f31a2de2fcad0ff104f56845777567a543f19157f772d7bc"}
May 26 16:11:43.270685 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:43.270682    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cmqx6w"
May 26 16:11:43.270685 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:43.270689    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c13c6af8075d816f31a2de2fcad0ff104f56845777567a543f19157f772d7bc"
May 26 16:11:45.506643 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.506611    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"]
May 26 16:11:45.506990 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.506979    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerName="extract"
May 26 16:11:45.507034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.506992    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerName="extract"
May 26 16:11:45.507034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.507010    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerName="util"
May 26 16:11:45.507034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.507015    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerName="util"
May 26 16:11:45.507034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.507023    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerName="pull"
May 26 16:11:45.507034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.507029    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerName="pull"
May 26 16:11:45.507191 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.507079    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="0da4fcc8-2459-40c3-bab7-74ce418d83b5" containerName="extract"
May 26 16:11:45.511153 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.511133    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:11:45.513355 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.513332    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 26 16:11:45.513459 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.513352    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 26 16:11:45.513459 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.513341    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 26 16:11:45.513459 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.513421    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-8c7qx\""
May 26 16:11:45.519262 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.519239    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"]
May 26 16:11:45.680271 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.680240    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/63b0a313-bb46-4f86-9300-3e1eded5a089-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-49jlz\" (UID: \"63b0a313-bb46-4f86-9300-3e1eded5a089\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:11:45.680440 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.680329    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76kwt\" (UniqueName: \"kubernetes.io/projected/63b0a313-bb46-4f86-9300-3e1eded5a089-kube-api-access-76kwt\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-49jlz\" (UID: \"63b0a313-bb46-4f86-9300-3e1eded5a089\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:11:45.781605 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.781537    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/63b0a313-bb46-4f86-9300-3e1eded5a089-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-49jlz\" (UID: \"63b0a313-bb46-4f86-9300-3e1eded5a089\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:11:45.781605 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.781593    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-76kwt\" (UniqueName: \"kubernetes.io/projected/63b0a313-bb46-4f86-9300-3e1eded5a089-kube-api-access-76kwt\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-49jlz\" (UID: \"63b0a313-bb46-4f86-9300-3e1eded5a089\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:11:45.783871 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.783848    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/63b0a313-bb46-4f86-9300-3e1eded5a089-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-49jlz\" (UID: \"63b0a313-bb46-4f86-9300-3e1eded5a089\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:11:45.788841 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.788817    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kwt\" (UniqueName: \"kubernetes.io/projected/63b0a313-bb46-4f86-9300-3e1eded5a089-kube-api-access-76kwt\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-49jlz\" (UID: \"63b0a313-bb46-4f86-9300-3e1eded5a089\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:11:45.821558 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.821536    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:11:45.949193 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:45.949165    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"]
May 26 16:11:45.952655 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:11:45.952625    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b0a313_bb46_4f86_9300_3e1eded5a089.slice/crio-b8ac6720e4fdb5e995ab84390fda5a13f6e98803db07c1b6591a7e9236ce5509 WatchSource:0}: Error finding container b8ac6720e4fdb5e995ab84390fda5a13f6e98803db07c1b6591a7e9236ce5509: Status 404 returned error can't find the container with id b8ac6720e4fdb5e995ab84390fda5a13f6e98803db07c1b6591a7e9236ce5509
May 26 16:11:46.280327 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:46.280298    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz" event={"ID":"63b0a313-bb46-4f86-9300-3e1eded5a089","Type":"ContainerStarted","Data":"b8ac6720e4fdb5e995ab84390fda5a13f6e98803db07c1b6591a7e9236ce5509"}
May 26 16:11:51.988498 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:51.988466    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-fxbks"]
May 26 16:11:51.990563 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:51.990545    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:51.993561 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:51.993539    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 26 16:11:51.993833 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:51.993818    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-q5xzz\""
May 26 16:11:51.995455 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:51.995439    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 26 16:11:52.002165 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.002144    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-fxbks"]
May 26 16:11:52.138617 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.138583    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:52.138794 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.138659    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4tvw\" (UniqueName: \"kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-kube-api-access-g4tvw\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:52.138794 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.138692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f3644e8f-f44d-4455-bb94-62b34caac664-cabundle0\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:52.239193 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.239108    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:52.239193 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.239189    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g4tvw\" (UniqueName: \"kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-kube-api-access-g4tvw\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:52.239411 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.239228    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f3644e8f-f44d-4455-bb94-62b34caac664-cabundle0\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:52.239411 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.239269    2570 secret.go:281] references non-existent secret key: ca.crt
May 26 16:11:52.239411 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.239310    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 16:11:52.239411 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.239321    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fxbks: references non-existent secret key: ca.crt
May 26 16:11:52.239411 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.239393    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates podName:f3644e8f-f44d-4455-bb94-62b34caac664 nodeName:}" failed. No retries permitted until 2026-05-26 16:11:52.739372116 +0000 UTC m=+664.120694120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates") pod "keda-operator-ffbb595cb-fxbks" (UID: "f3644e8f-f44d-4455-bb94-62b34caac664") : references non-existent secret key: ca.crt
May 26 16:11:52.239880 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.239860    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f3644e8f-f44d-4455-bb94-62b34caac664-cabundle0\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:52.249766 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.249739    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4tvw\" (UniqueName: \"kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-kube-api-access-g4tvw\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:52.309763 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.309709    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz" event={"ID":"63b0a313-bb46-4f86-9300-3e1eded5a089","Type":"ContainerStarted","Data":"587be7444eac7c816858b176d9bffb1d13bdf8aaca9dedf0150a5f45d4963304"}
May 26 16:11:52.309913 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.309799    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:11:52.319580 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.319547    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"]
May 26 16:11:52.321778 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.321763    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:52.323910 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.323891    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 26 16:11:52.330972 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.330951    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"]
May 26 16:11:52.332531 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.332495    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz" podStartSLOduration=1.828277505 podStartE2EDuration="7.332483668s" podCreationTimestamp="2026-05-26 16:11:45 +0000 UTC" firstStartedPulling="2026-05-26 16:11:45.954514971 +0000 UTC m=+657.335836977" lastFinishedPulling="2026-05-26 16:11:51.458721138 +0000 UTC m=+662.840043140" observedRunningTime="2026-05-26 16:11:52.33183149 +0000 UTC m=+663.713153516" watchObservedRunningTime="2026-05-26 16:11:52.332483668 +0000 UTC m=+663.713805694"
May 26 16:11:52.441062 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.441027    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:52.441305 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.441078    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85xml\" (UniqueName: \"kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-kube-api-access-85xml\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:52.441305 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.441234    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/741eafc2-2151-42cc-ab99-0ca7bc6936c6-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:52.542343 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.542241    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/741eafc2-2151-42cc-ab99-0ca7bc6936c6-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:52.542488 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.542374    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:52.542488 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.542425    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-85xml\" (UniqueName: \"kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-kube-api-access-85xml\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:52.542606 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.542568    2570 secret.go:281] references non-existent secret key: tls.crt
May 26 16:11:52.542606 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.542583    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:11:52.542606 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.542600    2570 projected.go:264] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
May 26 16:11:52.542747 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.542622    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 26 16:11:52.542747 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.542671    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates podName:741eafc2-2151-42cc-ab99-0ca7bc6936c6 nodeName:}" failed. No retries permitted until 2026-05-26 16:11:53.042654482 +0000 UTC m=+664.423976499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates") pod "keda-metrics-apiserver-7c9f485588-rfqp5" (UID: "741eafc2-2151-42cc-ab99-0ca7bc6936c6") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 26 16:11:52.544088 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.543033    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/741eafc2-2151-42cc-ab99-0ca7bc6936c6-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:52.569735 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.569708    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85xml\" (UniqueName: \"kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-kube-api-access-85xml\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:52.744961 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:52.744918    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:52.745119 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.745064    2570 secret.go:281] references non-existent secret key: ca.crt
May 26 16:11:52.745119 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.745086    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 16:11:52.745119 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.745097    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fxbks: references non-existent secret key: ca.crt
May 26 16:11:52.745229 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:52.745161    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates podName:f3644e8f-f44d-4455-bb94-62b34caac664 nodeName:}" failed. No retries permitted until 2026-05-26 16:11:53.745139966 +0000 UTC m=+665.126461973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates") pod "keda-operator-ffbb595cb-fxbks" (UID: "f3644e8f-f44d-4455-bb94-62b34caac664") : references non-existent secret key: ca.crt
May 26 16:11:53.047514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:53.047480    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:53.047886 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:53.047649    2570 secret.go:281] references non-existent secret key: tls.crt
May 26 16:11:53.047886 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:53.047673    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:11:53.047886 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:53.047697    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5: references non-existent secret key: tls.crt
May 26 16:11:53.047886 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:53.047759    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates podName:741eafc2-2151-42cc-ab99-0ca7bc6936c6 nodeName:}" failed. No retries permitted until 2026-05-26 16:11:54.047739576 +0000 UTC m=+665.429061585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates") pod "keda-metrics-apiserver-7c9f485588-rfqp5" (UID: "741eafc2-2151-42cc-ab99-0ca7bc6936c6") : references non-existent secret key: tls.crt
May 26 16:11:53.753760 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:53.753716    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:53.753956 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:53.753835    2570 secret.go:281] references non-existent secret key: ca.crt
May 26 16:11:53.753956 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:53.753854    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 16:11:53.753956 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:53.753863    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fxbks: references non-existent secret key: ca.crt
May 26 16:11:53.753956 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:53.753921    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates podName:f3644e8f-f44d-4455-bb94-62b34caac664 nodeName:}" failed. No retries permitted until 2026-05-26 16:11:55.753903412 +0000 UTC m=+667.135225416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates") pod "keda-operator-ffbb595cb-fxbks" (UID: "f3644e8f-f44d-4455-bb94-62b34caac664") : references non-existent secret key: ca.crt
May 26 16:11:54.056843 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:54.056748    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:54.057258 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:54.056904    2570 secret.go:281] references non-existent secret key: tls.crt
May 26 16:11:54.057258 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:54.056928    2570 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:11:54.057258 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:54.056963    2570 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5: references non-existent secret key: tls.crt
May 26 16:11:54.057258 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:11:54.057041    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates podName:741eafc2-2151-42cc-ab99-0ca7bc6936c6 nodeName:}" failed. No retries permitted until 2026-05-26 16:11:56.057021013 +0000 UTC m=+667.438343021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates") pod "keda-metrics-apiserver-7c9f485588-rfqp5" (UID: "741eafc2-2151-42cc-ab99-0ca7bc6936c6") : references non-existent secret key: tls.crt
May 26 16:11:55.769831 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:55.769794    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:55.772147 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:55.772124    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3644e8f-f44d-4455-bb94-62b34caac664-certificates\") pod \"keda-operator-ffbb595cb-fxbks\" (UID: \"f3644e8f-f44d-4455-bb94-62b34caac664\") " pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:55.901883 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:55.901838    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:11:56.033674 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:56.033601    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-fxbks"]
May 26 16:11:56.036447 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:11:56.036418    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3644e8f_f44d_4455_bb94_62b34caac664.slice/crio-bbdca24691629b737287e3d43c6c02e66b6ce9651dd128bf08418a784b51ae2f WatchSource:0}: Error finding container bbdca24691629b737287e3d43c6c02e66b6ce9651dd128bf08418a784b51ae2f: Status 404 returned error can't find the container with id bbdca24691629b737287e3d43c6c02e66b6ce9651dd128bf08418a784b51ae2f
May 26 16:11:56.072398 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:56.072370    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:56.074930 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:56.074911    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/741eafc2-2151-42cc-ab99-0ca7bc6936c6-certificates\") pod \"keda-metrics-apiserver-7c9f485588-rfqp5\" (UID: \"741eafc2-2151-42cc-ab99-0ca7bc6936c6\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:56.232243 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:56.232208    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:11:56.326068 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:56.326033    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-fxbks" event={"ID":"f3644e8f-f44d-4455-bb94-62b34caac664","Type":"ContainerStarted","Data":"bbdca24691629b737287e3d43c6c02e66b6ce9651dd128bf08418a784b51ae2f"}
May 26 16:11:56.361505 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:56.361480    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"]
May 26 16:11:56.363575 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:11:56.363550    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741eafc2_2151_42cc_ab99_0ca7bc6936c6.slice/crio-ac7836e00e0f659b91044e1e3359ea05a27fab34e030006caa1bac05f515b9cb WatchSource:0}: Error finding container ac7836e00e0f659b91044e1e3359ea05a27fab34e030006caa1bac05f515b9cb: Status 404 returned error can't find the container with id ac7836e00e0f659b91044e1e3359ea05a27fab34e030006caa1bac05f515b9cb
May 26 16:11:57.330981 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:11:57.330939    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5" event={"ID":"741eafc2-2151-42cc-ab99-0ca7bc6936c6","Type":"ContainerStarted","Data":"ac7836e00e0f659b91044e1e3359ea05a27fab34e030006caa1bac05f515b9cb"}
May 26 16:12:01.346502 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:01.346465    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5" event={"ID":"741eafc2-2151-42cc-ab99-0ca7bc6936c6","Type":"ContainerStarted","Data":"c730a13a97b58638d24bb0b4f332e6cfb79dfae62aa05ffdd885de8011a0f3dc"}
May 26 16:12:01.346935 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:01.346559    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:12:01.347785 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:01.347763    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-fxbks" event={"ID":"f3644e8f-f44d-4455-bb94-62b34caac664","Type":"ContainerStarted","Data":"0100da4907c4ba425f0377ff16f10b604cec405ad640f187b07da31685adbdba"}
May 26 16:12:01.347945 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:01.347929    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:12:01.364892 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:01.364849    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5" podStartSLOduration=5.204906658 podStartE2EDuration="9.364836199s" podCreationTimestamp="2026-05-26 16:11:52 +0000 UTC" firstStartedPulling="2026-05-26 16:11:56.364882183 +0000 UTC m=+667.746204188" lastFinishedPulling="2026-05-26 16:12:00.524811708 +0000 UTC m=+671.906133729" observedRunningTime="2026-05-26 16:12:01.363503003 +0000 UTC m=+672.744825029" watchObservedRunningTime="2026-05-26 16:12:01.364836199 +0000 UTC m=+672.746158224"
May 26 16:12:01.383484 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:01.383435    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-fxbks" podStartSLOduration=5.892573821 podStartE2EDuration="10.383422968s" podCreationTimestamp="2026-05-26 16:11:51 +0000 UTC" firstStartedPulling="2026-05-26 16:11:56.037800375 +0000 UTC m=+667.419122391" lastFinishedPulling="2026-05-26 16:12:00.528649531 +0000 UTC m=+671.909971538" observedRunningTime="2026-05-26 16:12:01.38109918 +0000 UTC m=+672.762421204" watchObservedRunningTime="2026-05-26 16:12:01.383422968 +0000 UTC m=+672.764744994"
May 26 16:12:12.355890 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:12.355861    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-rfqp5"
May 26 16:12:13.315939 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:13.315910    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-49jlz"
May 26 16:12:22.354095 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:22.354061    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-fxbks"
May 26 16:12:56.472265 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.472187    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"]
May 26 16:12:56.474623 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.474608    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:12:56.476987 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.476963    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 26 16:12:56.477230 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.476996    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 16:12:56.477230 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.477001    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 16:12:56.477422 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.477047    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-bbnzk\""
May 26 16:12:56.486998 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.486976    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"]
May 26 16:12:56.491854 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.491829    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk4gt\" (UniqueName: \"kubernetes.io/projected/cdb1320c-d47a-4e45-b0c3-1953a35cb52e-kube-api-access-vk4gt\") pod \"llmisvc-controller-manager-6b94ff949c-9dqgq\" (UID: \"cdb1320c-d47a-4e45-b0c3-1953a35cb52e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:12:56.491965 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.491902    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdb1320c-d47a-4e45-b0c3-1953a35cb52e-cert\") pod \"llmisvc-controller-manager-6b94ff949c-9dqgq\" (UID: \"cdb1320c-d47a-4e45-b0c3-1953a35cb52e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:12:56.492388 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.492371    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-mmjxf"]
May 26 16:12:56.494524 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.494506    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:12:56.496590 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.496572    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gv98l\""
May 26 16:12:56.496688 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.496588    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 16:12:56.504919 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.504897    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-mmjxf"]
May 26 16:12:56.592753 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.592716    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7dg\" (UniqueName: \"kubernetes.io/projected/00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c-kube-api-access-pc7dg\") pod \"seaweedfs-79f457656b-mmjxf\" (UID: \"00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c\") " pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:12:56.592928 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.592778    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vk4gt\" (UniqueName: \"kubernetes.io/projected/cdb1320c-d47a-4e45-b0c3-1953a35cb52e-kube-api-access-vk4gt\") pod \"llmisvc-controller-manager-6b94ff949c-9dqgq\" (UID: \"cdb1320c-d47a-4e45-b0c3-1953a35cb52e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:12:56.592928 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.592827    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c-data\") pod \"seaweedfs-79f457656b-mmjxf\" (UID: \"00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c\") " pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:12:56.592928 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.592882    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdb1320c-d47a-4e45-b0c3-1953a35cb52e-cert\") pod \"llmisvc-controller-manager-6b94ff949c-9dqgq\" (UID: \"cdb1320c-d47a-4e45-b0c3-1953a35cb52e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:12:56.593029 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:12:56.592985    2570 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
May 26 16:12:56.593071 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:12:56.593058    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdb1320c-d47a-4e45-b0c3-1953a35cb52e-cert podName:cdb1320c-d47a-4e45-b0c3-1953a35cb52e nodeName:}" failed. No retries permitted until 2026-05-26 16:12:57.093037435 +0000 UTC m=+728.474359442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cdb1320c-d47a-4e45-b0c3-1953a35cb52e-cert") pod "llmisvc-controller-manager-6b94ff949c-9dqgq" (UID: "cdb1320c-d47a-4e45-b0c3-1953a35cb52e") : secret "llmisvc-webhook-server-cert" not found
May 26 16:12:56.605996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.605974    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk4gt\" (UniqueName: \"kubernetes.io/projected/cdb1320c-d47a-4e45-b0c3-1953a35cb52e-kube-api-access-vk4gt\") pod \"llmisvc-controller-manager-6b94ff949c-9dqgq\" (UID: \"cdb1320c-d47a-4e45-b0c3-1953a35cb52e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:12:56.693261 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.693227    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7dg\" (UniqueName: \"kubernetes.io/projected/00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c-kube-api-access-pc7dg\") pod \"seaweedfs-79f457656b-mmjxf\" (UID: \"00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c\") " pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:12:56.693442 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.693319    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c-data\") pod \"seaweedfs-79f457656b-mmjxf\" (UID: \"00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c\") " pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:12:56.693751 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.693731    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c-data\") pod \"seaweedfs-79f457656b-mmjxf\" (UID: \"00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c\") " pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:12:56.703090 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.703064    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7dg\" (UniqueName: \"kubernetes.io/projected/00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c-kube-api-access-pc7dg\") pod \"seaweedfs-79f457656b-mmjxf\" (UID: \"00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c\") " pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:12:56.803936 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.803859    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:12:56.935312 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:56.935274    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-mmjxf"]
May 26 16:12:56.937452 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:12:56.937423    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d6a42d_2d5f_465c_ab2f_0195f5ddbc0c.slice/crio-91976766e603c6bf10d1dfacb964e435af69a0c8a7e59345c87bd5253bf4d9ab WatchSource:0}: Error finding container 91976766e603c6bf10d1dfacb964e435af69a0c8a7e59345c87bd5253bf4d9ab: Status 404 returned error can't find the container with id 91976766e603c6bf10d1dfacb964e435af69a0c8a7e59345c87bd5253bf4d9ab
May 26 16:12:57.095879 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:57.095798    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdb1320c-d47a-4e45-b0c3-1953a35cb52e-cert\") pod \"llmisvc-controller-manager-6b94ff949c-9dqgq\" (UID: \"cdb1320c-d47a-4e45-b0c3-1953a35cb52e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:12:57.098195 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:57.098165    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdb1320c-d47a-4e45-b0c3-1953a35cb52e-cert\") pod \"llmisvc-controller-manager-6b94ff949c-9dqgq\" (UID: \"cdb1320c-d47a-4e45-b0c3-1953a35cb52e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:12:57.385852 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:57.385822    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:12:57.513041 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:57.512906    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"]
May 26 16:12:57.515007 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:12:57.514975    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcdb1320c_d47a_4e45_b0c3_1953a35cb52e.slice/crio-91d02ad53496c78454014bf638b9a254e1547628f2de0938d926b04e64cc46d0 WatchSource:0}: Error finding container 91d02ad53496c78454014bf638b9a254e1547628f2de0938d926b04e64cc46d0: Status 404 returned error can't find the container with id 91d02ad53496c78454014bf638b9a254e1547628f2de0938d926b04e64cc46d0
May 26 16:12:57.532939 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:57.532905    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq" event={"ID":"cdb1320c-d47a-4e45-b0c3-1953a35cb52e","Type":"ContainerStarted","Data":"91d02ad53496c78454014bf638b9a254e1547628f2de0938d926b04e64cc46d0"}
May 26 16:12:57.533828 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:12:57.533809    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-mmjxf" event={"ID":"00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c","Type":"ContainerStarted","Data":"91976766e603c6bf10d1dfacb964e435af69a0c8a7e59345c87bd5253bf4d9ab"}
May 26 16:13:01.549910 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:13:01.549872    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq" event={"ID":"cdb1320c-d47a-4e45-b0c3-1953a35cb52e","Type":"ContainerStarted","Data":"c8b9230a0b188d4646717453fb881395b6fa588fb2bbf3839f880bf66259f8b9"}
May 26 16:13:01.550417 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:13:01.549989    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:13:01.551618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:13:01.551569    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-mmjxf" event={"ID":"00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c","Type":"ContainerStarted","Data":"24138be69dd8fe959eafcdcd222ff4bc2c1e828f6d59c5d3fc46d2209c938bbd"}
May 26 16:13:01.551871 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:13:01.551840    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:13:01.566475 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:13:01.566428    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq" podStartSLOduration=2.289988687 podStartE2EDuration="5.566413563s" podCreationTimestamp="2026-05-26 16:12:56 +0000 UTC" firstStartedPulling="2026-05-26 16:12:57.516334774 +0000 UTC m=+728.897656781" lastFinishedPulling="2026-05-26 16:13:00.792759651 +0000 UTC m=+732.174081657" observedRunningTime="2026-05-26 16:13:01.564104809 +0000 UTC m=+732.945426834" watchObservedRunningTime="2026-05-26 16:13:01.566413563 +0000 UTC m=+732.947735587"
May 26 16:13:01.578444 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:13:01.578395    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-mmjxf" podStartSLOduration=1.673915635 podStartE2EDuration="5.57837915s" podCreationTimestamp="2026-05-26 16:12:56 +0000 UTC" firstStartedPulling="2026-05-26 16:12:56.938611677 +0000 UTC m=+728.319933681" lastFinishedPulling="2026-05-26 16:13:00.84307519 +0000 UTC m=+732.224397196" observedRunningTime="2026-05-26 16:13:01.577405182 +0000 UTC m=+732.958727208" watchObservedRunningTime="2026-05-26 16:13:01.57837915 +0000 UTC m=+732.959701175"
May 26 16:13:07.557843 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:13:07.557809    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-mmjxf"
May 26 16:13:32.557836 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:13:32.557805    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-9dqgq"
May 26 16:14:08.005620 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.005587    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-6nklx"]
May 26 16:14:08.008880 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.008860    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:08.010940 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.010914    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 26 16:14:08.011050 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.011012    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-pxqrx\""
May 26 16:14:08.018029 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.018007    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-6nklx"]
May 26 16:14:08.123273 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.123232    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1c8d171-8b19-4e7e-804d-e0e8fda9f101-cert\") pod \"odh-model-controller-696fc77849-6nklx\" (UID: \"c1c8d171-8b19-4e7e-804d-e0e8fda9f101\") " pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:08.123505 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.123368    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn8d9\" (UniqueName: \"kubernetes.io/projected/c1c8d171-8b19-4e7e-804d-e0e8fda9f101-kube-api-access-gn8d9\") pod \"odh-model-controller-696fc77849-6nklx\" (UID: \"c1c8d171-8b19-4e7e-804d-e0e8fda9f101\") " pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:08.224765 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.224724    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1c8d171-8b19-4e7e-804d-e0e8fda9f101-cert\") pod \"odh-model-controller-696fc77849-6nklx\" (UID: \"c1c8d171-8b19-4e7e-804d-e0e8fda9f101\") " pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:08.224950 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.224801    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gn8d9\" (UniqueName: \"kubernetes.io/projected/c1c8d171-8b19-4e7e-804d-e0e8fda9f101-kube-api-access-gn8d9\") pod \"odh-model-controller-696fc77849-6nklx\" (UID: \"c1c8d171-8b19-4e7e-804d-e0e8fda9f101\") " pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:08.224950 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:14:08.224885    2570 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
May 26 16:14:08.225058 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:14:08.224964    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c8d171-8b19-4e7e-804d-e0e8fda9f101-cert podName:c1c8d171-8b19-4e7e-804d-e0e8fda9f101 nodeName:}" failed. No retries permitted until 2026-05-26 16:14:08.724945945 +0000 UTC m=+800.106267953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c1c8d171-8b19-4e7e-804d-e0e8fda9f101-cert") pod "odh-model-controller-696fc77849-6nklx" (UID: "c1c8d171-8b19-4e7e-804d-e0e8fda9f101") : secret "odh-model-controller-webhook-cert" not found
May 26 16:14:08.237372 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.237343    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn8d9\" (UniqueName: \"kubernetes.io/projected/c1c8d171-8b19-4e7e-804d-e0e8fda9f101-kube-api-access-gn8d9\") pod \"odh-model-controller-696fc77849-6nklx\" (UID: \"c1c8d171-8b19-4e7e-804d-e0e8fda9f101\") " pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:08.729042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.728954    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1c8d171-8b19-4e7e-804d-e0e8fda9f101-cert\") pod \"odh-model-controller-696fc77849-6nklx\" (UID: \"c1c8d171-8b19-4e7e-804d-e0e8fda9f101\") " pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:08.731383 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.731351    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1c8d171-8b19-4e7e-804d-e0e8fda9f101-cert\") pod \"odh-model-controller-696fc77849-6nklx\" (UID: \"c1c8d171-8b19-4e7e-804d-e0e8fda9f101\") " pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:08.919722 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:08.919679    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:09.043494 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:09.043435    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-6nklx"]
May 26 16:14:09.045688 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:14:09.045662    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c8d171_8b19_4e7e_804d_e0e8fda9f101.slice/crio-aa7d2f1ce1bf9bc6f5d43b717988332c821ce5763adf6afff0d6e95dc5d9049c WatchSource:0}: Error finding container aa7d2f1ce1bf9bc6f5d43b717988332c821ce5763adf6afff0d6e95dc5d9049c: Status 404 returned error can't find the container with id aa7d2f1ce1bf9bc6f5d43b717988332c821ce5763adf6afff0d6e95dc5d9049c
May 26 16:14:09.776151 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:09.776112    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-6nklx" event={"ID":"c1c8d171-8b19-4e7e-804d-e0e8fda9f101","Type":"ContainerStarted","Data":"aa7d2f1ce1bf9bc6f5d43b717988332c821ce5763adf6afff0d6e95dc5d9049c"}
May 26 16:14:11.784870 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:11.784829    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-6nklx" event={"ID":"c1c8d171-8b19-4e7e-804d-e0e8fda9f101","Type":"ContainerStarted","Data":"a9f965337c22d3ee1a12de29e138837979c26088d27431a4993df1ff2088bb46"}
May 26 16:14:11.785418 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:11.785043    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:11.800359 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:11.800306    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-6nklx" podStartSLOduration=2.186815368 podStartE2EDuration="4.800269706s" podCreationTimestamp="2026-05-26 16:14:07 +0000 UTC" firstStartedPulling="2026-05-26 16:14:09.046911134 +0000 UTC m=+800.428233140" lastFinishedPulling="2026-05-26 16:14:11.660365458 +0000 UTC m=+803.041687478" observedRunningTime="2026-05-26 16:14:11.798922444 +0000 UTC m=+803.180244472" watchObservedRunningTime="2026-05-26 16:14:11.800269706 +0000 UTC m=+803.181591730"
May 26 16:14:22.791275 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:22.791237    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-6nklx"
May 26 16:14:45.683805 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.683772    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"]
May 26 16:14:45.687918 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.687895    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.690204 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.690182    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dzvrr\""
May 26 16:14:45.690324 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.690217    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c73f1-predictor-serving-cert\""
May 26 16:14:45.691054 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.691035    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:14:45.691151 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.691070    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:14:45.691151 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.691038    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c73f1-kube-rbac-proxy-sar-config\""
May 26 16:14:45.697311 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.697263    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"]
May 26 16:14:45.754123 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.754083    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvrd\" (UniqueName: \"kubernetes.io/projected/aa96e43d-98b3-4804-bf65-e705013a2e3c-kube-api-access-vwvrd\") pod \"success-200-isvc-c73f1-predictor-9985b79f8-6gnkh\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") " pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.754273 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.754209    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa96e43d-98b3-4804-bf65-e705013a2e3c-proxy-tls\") pod \"success-200-isvc-c73f1-predictor-9985b79f8-6gnkh\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") " pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.754273 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.754248    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa96e43d-98b3-4804-bf65-e705013a2e3c-success-200-isvc-c73f1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c73f1-predictor-9985b79f8-6gnkh\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") " pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.854840 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.854803    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa96e43d-98b3-4804-bf65-e705013a2e3c-proxy-tls\") pod \"success-200-isvc-c73f1-predictor-9985b79f8-6gnkh\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") " pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.855015 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.854846    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa96e43d-98b3-4804-bf65-e705013a2e3c-success-200-isvc-c73f1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c73f1-predictor-9985b79f8-6gnkh\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") " pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.855015 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.854915    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvrd\" (UniqueName: \"kubernetes.io/projected/aa96e43d-98b3-4804-bf65-e705013a2e3c-kube-api-access-vwvrd\") pod \"success-200-isvc-c73f1-predictor-9985b79f8-6gnkh\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") " pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.855612 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.855584    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa96e43d-98b3-4804-bf65-e705013a2e3c-success-200-isvc-c73f1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c73f1-predictor-9985b79f8-6gnkh\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") " pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.857272 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.857252    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa96e43d-98b3-4804-bf65-e705013a2e3c-proxy-tls\") pod \"success-200-isvc-c73f1-predictor-9985b79f8-6gnkh\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") " pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.865113 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.865090    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvrd\" (UniqueName: \"kubernetes.io/projected/aa96e43d-98b3-4804-bf65-e705013a2e3c-kube-api-access-vwvrd\") pod \"success-200-isvc-c73f1-predictor-9985b79f8-6gnkh\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") " pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:45.998947 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:45.998869    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:14:46.135786 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:46.135759    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"]
May 26 16:14:46.137753 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:14:46.137723    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa96e43d_98b3_4804_bf65_e705013a2e3c.slice/crio-644e8b91e2d5979c0ecc92a99ad90db5150d74c4137fe06ffe8fef1fa8b1c96b WatchSource:0}: Error finding container 644e8b91e2d5979c0ecc92a99ad90db5150d74c4137fe06ffe8fef1fa8b1c96b: Status 404 returned error can't find the container with id 644e8b91e2d5979c0ecc92a99ad90db5150d74c4137fe06ffe8fef1fa8b1c96b
May 26 16:14:46.914885 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:46.914842    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" event={"ID":"aa96e43d-98b3-4804-bf65-e705013a2e3c","Type":"ContainerStarted","Data":"644e8b91e2d5979c0ecc92a99ad90db5150d74c4137fe06ffe8fef1fa8b1c96b"}
May 26 16:14:57.974367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:57.974270    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:14:58.967015 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:14:58.966972    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" event={"ID":"aa96e43d-98b3-4804-bf65-e705013a2e3c","Type":"ContainerStarted","Data":"cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c"}
May 26 16:15:00.975209 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:00.975171    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" event={"ID":"aa96e43d-98b3-4804-bf65-e705013a2e3c","Type":"ContainerStarted","Data":"9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a"}
May 26 16:15:00.975589 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:00.975268    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:15:00.995491 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:00.995441    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podStartSLOduration=1.8745601170000001 podStartE2EDuration="15.995426543s" podCreationTimestamp="2026-05-26 16:14:45 +0000 UTC" firstStartedPulling="2026-05-26 16:14:46.139602989 +0000 UTC m=+837.520924995" lastFinishedPulling="2026-05-26 16:15:00.260469411 +0000 UTC m=+851.641791421" observedRunningTime="2026-05-26 16:15:00.994135794 +0000 UTC m=+852.375457821" watchObservedRunningTime="2026-05-26 16:15:00.995426543 +0000 UTC m=+852.376748589"
May 26 16:15:01.979010 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:01.978975    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:15:01.980288 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:01.980247    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 16:15:02.982731 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:02.982686    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 16:15:07.987080 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:07.987045    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:15:07.987675 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:07.987645    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 16:15:17.987594 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:17.987552    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 16:15:27.988218 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:27.988173    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 16:15:37.988260 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:37.988215    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 16:15:47.988988 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:15:47.988919    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:16:05.790556 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.790519    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"]
May 26 16:16:05.793960 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.793943    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:05.795892 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.795860    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c73f1-kube-rbac-proxy-sar-config\""
May 26 16:16:05.795892 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.795873    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c73f1-serving-cert\""
May 26 16:16:05.802204 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.802178    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"]
May 26 16:16:05.889256 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.889220    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e437aae-3787-4015-bd15-6d58f1c88265-proxy-tls\") pod \"switch-graph-c73f1-864fbbd8d6-ns5f9\" (UID: \"0e437aae-3787-4015-bd15-6d58f1c88265\") " pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:05.889453 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.889272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e437aae-3787-4015-bd15-6d58f1c88265-openshift-service-ca-bundle\") pod \"switch-graph-c73f1-864fbbd8d6-ns5f9\" (UID: \"0e437aae-3787-4015-bd15-6d58f1c88265\") " pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:05.990148 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.990111    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e437aae-3787-4015-bd15-6d58f1c88265-proxy-tls\") pod \"switch-graph-c73f1-864fbbd8d6-ns5f9\" (UID: \"0e437aae-3787-4015-bd15-6d58f1c88265\") " pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:05.990148 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.990155    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e437aae-3787-4015-bd15-6d58f1c88265-openshift-service-ca-bundle\") pod \"switch-graph-c73f1-864fbbd8d6-ns5f9\" (UID: \"0e437aae-3787-4015-bd15-6d58f1c88265\") " pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:05.990795 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.990775    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e437aae-3787-4015-bd15-6d58f1c88265-openshift-service-ca-bundle\") pod \"switch-graph-c73f1-864fbbd8d6-ns5f9\" (UID: \"0e437aae-3787-4015-bd15-6d58f1c88265\") " pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:05.992718 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:05.992693    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e437aae-3787-4015-bd15-6d58f1c88265-proxy-tls\") pod \"switch-graph-c73f1-864fbbd8d6-ns5f9\" (UID: \"0e437aae-3787-4015-bd15-6d58f1c88265\") " pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:06.105404 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:06.105322    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:06.235064 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:06.235032    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"]
May 26 16:16:06.238519 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:16:06.238492    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e437aae_3787_4015_bd15_6d58f1c88265.slice/crio-5fb1654c49653b332b493265ef16816d3206e5600286a58a187886a502651ee4 WatchSource:0}: Error finding container 5fb1654c49653b332b493265ef16816d3206e5600286a58a187886a502651ee4: Status 404 returned error can't find the container with id 5fb1654c49653b332b493265ef16816d3206e5600286a58a187886a502651ee4
May 26 16:16:07.206012 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:07.205923    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" event={"ID":"0e437aae-3787-4015-bd15-6d58f1c88265","Type":"ContainerStarted","Data":"5fb1654c49653b332b493265ef16816d3206e5600286a58a187886a502651ee4"}
May 26 16:16:09.214132 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:09.214093    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" event={"ID":"0e437aae-3787-4015-bd15-6d58f1c88265","Type":"ContainerStarted","Data":"6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca"}
May 26 16:16:09.214509 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:09.214235    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:09.230581 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:09.230538    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" podStartSLOduration=1.8425801069999999 podStartE2EDuration="4.230524385s" podCreationTimestamp="2026-05-26 16:16:05 +0000 UTC" firstStartedPulling="2026-05-26 16:16:06.240274037 +0000 UTC m=+917.621596044" lastFinishedPulling="2026-05-26 16:16:08.628218317 +0000 UTC m=+920.009540322" observedRunningTime="2026-05-26 16:16:09.228565028 +0000 UTC m=+920.609887053" watchObservedRunningTime="2026-05-26 16:16:09.230524385 +0000 UTC m=+920.611846464"
May 26 16:16:15.223637 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:15.223609    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:15.990977 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:15.990941    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"]
May 26 16:16:15.991173 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:15.991134    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1" containerID="cri-o://6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca" gracePeriod=30
May 26 16:16:16.114911 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.114879    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"]
May 26 16:16:16.115218 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.115190    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container" containerID="cri-o://cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c" gracePeriod=30
May 26 16:16:16.115352 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.115232    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kube-rbac-proxy" containerID="cri-o://9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a" gracePeriod=30
May 26 16:16:16.157403 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.157373    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"]
May 26 16:16:16.160912 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.160897    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.162795 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.162772    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-df0d2-predictor-serving-cert\""
May 26 16:16:16.162795 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.162784    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-df0d2-kube-rbac-proxy-sar-config\""
May 26 16:16:16.168312 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.168258    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"]
May 26 16:16:16.240407 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.240378    2570 generic.go:358] "Generic (PLEG): container finished" podID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerID="9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a" exitCode=2
May 26 16:16:16.240750 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.240415    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" event={"ID":"aa96e43d-98b3-4804-bf65-e705013a2e3c","Type":"ContainerDied","Data":"9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a"}
May 26 16:16:16.285972 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.285892    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1d29b3a-27a7-4483-9a6a-afe511aebef2-proxy-tls\") pod \"success-200-isvc-df0d2-predictor-8fddf9f46-js5fq\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") " pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.286119 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.285959    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qd9z\" (UniqueName: \"kubernetes.io/projected/a1d29b3a-27a7-4483-9a6a-afe511aebef2-kube-api-access-8qd9z\") pod \"success-200-isvc-df0d2-predictor-8fddf9f46-js5fq\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") " pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.286119 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.286031    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1d29b3a-27a7-4483-9a6a-afe511aebef2-success-200-isvc-df0d2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df0d2-predictor-8fddf9f46-js5fq\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") " pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.386519 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.386480    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1d29b3a-27a7-4483-9a6a-afe511aebef2-proxy-tls\") pod \"success-200-isvc-df0d2-predictor-8fddf9f46-js5fq\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") " pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.386725 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.386700    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8qd9z\" (UniqueName: \"kubernetes.io/projected/a1d29b3a-27a7-4483-9a6a-afe511aebef2-kube-api-access-8qd9z\") pod \"success-200-isvc-df0d2-predictor-8fddf9f46-js5fq\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") " pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.386920 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.386899    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1d29b3a-27a7-4483-9a6a-afe511aebef2-success-200-isvc-df0d2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df0d2-predictor-8fddf9f46-js5fq\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") " pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.387467 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.387445    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1d29b3a-27a7-4483-9a6a-afe511aebef2-success-200-isvc-df0d2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df0d2-predictor-8fddf9f46-js5fq\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") " pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.389175 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.389152    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1d29b3a-27a7-4483-9a6a-afe511aebef2-proxy-tls\") pod \"success-200-isvc-df0d2-predictor-8fddf9f46-js5fq\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") " pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.394567 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.394545    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qd9z\" (UniqueName: \"kubernetes.io/projected/a1d29b3a-27a7-4483-9a6a-afe511aebef2-kube-api-access-8qd9z\") pod \"success-200-isvc-df0d2-predictor-8fddf9f46-js5fq\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") " pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.472588 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.472551    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:16.601265 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:16.601239    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"]
May 26 16:16:16.603586 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:16:16.603556    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d29b3a_27a7_4483_9a6a_afe511aebef2.slice/crio-de762552845345bd628eeb4a4feeee0d0eccd0a7350012d2d5e9e250bdbf8c24 WatchSource:0}: Error finding container de762552845345bd628eeb4a4feeee0d0eccd0a7350012d2d5e9e250bdbf8c24: Status 404 returned error can't find the container with id de762552845345bd628eeb4a4feeee0d0eccd0a7350012d2d5e9e250bdbf8c24
May 26 16:16:17.245692 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:17.245654    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" event={"ID":"a1d29b3a-27a7-4483-9a6a-afe511aebef2","Type":"ContainerStarted","Data":"288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0"}
May 26 16:16:17.246076 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:17.245700    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" event={"ID":"a1d29b3a-27a7-4483-9a6a-afe511aebef2","Type":"ContainerStarted","Data":"e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05"}
May 26 16:16:17.246076 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:17.245713    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" event={"ID":"a1d29b3a-27a7-4483-9a6a-afe511aebef2","Type":"ContainerStarted","Data":"de762552845345bd628eeb4a4feeee0d0eccd0a7350012d2d5e9e250bdbf8c24"}
May 26 16:16:17.246076 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:17.245758    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:17.264392 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:17.264323    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podStartSLOduration=1.264310107 podStartE2EDuration="1.264310107s" podCreationTimestamp="2026-05-26 16:16:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:16:17.262263115 +0000 UTC m=+928.643585309" watchObservedRunningTime="2026-05-26 16:16:17.264310107 +0000 UTC m=+928.645632187"
May 26 16:16:17.983189 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:17.983149    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
May 26 16:16:17.987955 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:17.987923    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 26 16:16:18.249519 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:18.249429    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:18.250640 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:18.250613    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:16:19.158375 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.158353    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:16:19.253166 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.253091    2570 generic.go:358] "Generic (PLEG): container finished" podID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerID="cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c" exitCode=0
May 26 16:16:19.253166 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.253158    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"
May 26 16:16:19.253623 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.253175    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" event={"ID":"aa96e43d-98b3-4804-bf65-e705013a2e3c","Type":"ContainerDied","Data":"cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c"}
May 26 16:16:19.253623 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.253209    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh" event={"ID":"aa96e43d-98b3-4804-bf65-e705013a2e3c","Type":"ContainerDied","Data":"644e8b91e2d5979c0ecc92a99ad90db5150d74c4137fe06ffe8fef1fa8b1c96b"}
May 26 16:16:19.253623 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.253228    2570 scope.go:117] "RemoveContainer" containerID="9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a"
May 26 16:16:19.253623 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.253575    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:16:19.261084 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.261069    2570 scope.go:117] "RemoveContainer" containerID="cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c"
May 26 16:16:19.268010 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.267991    2570 scope.go:117] "RemoveContainer" containerID="9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a"
May 26 16:16:19.268247 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:16:19.268228    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a\": container with ID starting with 9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a not found: ID does not exist" containerID="9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a"
May 26 16:16:19.268339 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.268259    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a"} err="failed to get container status \"9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a\": rpc error: code = NotFound desc = could not find container \"9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a\": container with ID starting with 9d9b5ec44736fc78d5fd8082134d36e9a22793c08b46ccf1550fffe5fad68e7a not found: ID does not exist"
May 26 16:16:19.268339 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.268296    2570 scope.go:117] "RemoveContainer" containerID="cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c"
May 26 16:16:19.268557 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:16:19.268541    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c\": container with ID starting with cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c not found: ID does not exist" containerID="cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c"
May 26 16:16:19.268598 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.268564    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c"} err="failed to get container status \"cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c\": rpc error: code = NotFound desc = could not find container \"cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c\": container with ID starting with cdf5e9b2b7122a5fdee98f562e2a888fee5e8738cc1cdeafc1e3e8a74d47cb2c not found: ID does not exist"
May 26 16:16:19.313325 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.313300    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa96e43d-98b3-4804-bf65-e705013a2e3c-proxy-tls\") pod \"aa96e43d-98b3-4804-bf65-e705013a2e3c\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") "
May 26 16:16:19.313426 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.313333    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwvrd\" (UniqueName: \"kubernetes.io/projected/aa96e43d-98b3-4804-bf65-e705013a2e3c-kube-api-access-vwvrd\") pod \"aa96e43d-98b3-4804-bf65-e705013a2e3c\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") "
May 26 16:16:19.313426 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.313354    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa96e43d-98b3-4804-bf65-e705013a2e3c-success-200-isvc-c73f1-kube-rbac-proxy-sar-config\") pod \"aa96e43d-98b3-4804-bf65-e705013a2e3c\" (UID: \"aa96e43d-98b3-4804-bf65-e705013a2e3c\") "
May 26 16:16:19.313739 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.313719    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa96e43d-98b3-4804-bf65-e705013a2e3c-success-200-isvc-c73f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-c73f1-kube-rbac-proxy-sar-config") pod "aa96e43d-98b3-4804-bf65-e705013a2e3c" (UID: "aa96e43d-98b3-4804-bf65-e705013a2e3c"). InnerVolumeSpecName "success-200-isvc-c73f1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:16:19.315380 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.315352    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa96e43d-98b3-4804-bf65-e705013a2e3c-kube-api-access-vwvrd" (OuterVolumeSpecName: "kube-api-access-vwvrd") pod "aa96e43d-98b3-4804-bf65-e705013a2e3c" (UID: "aa96e43d-98b3-4804-bf65-e705013a2e3c"). InnerVolumeSpecName "kube-api-access-vwvrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:16:19.315484 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.315466    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa96e43d-98b3-4804-bf65-e705013a2e3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa96e43d-98b3-4804-bf65-e705013a2e3c" (UID: "aa96e43d-98b3-4804-bf65-e705013a2e3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:16:19.415156 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.415121    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa96e43d-98b3-4804-bf65-e705013a2e3c-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:16:19.415156 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.415149    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwvrd\" (UniqueName: \"kubernetes.io/projected/aa96e43d-98b3-4804-bf65-e705013a2e3c-kube-api-access-vwvrd\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:16:19.415156 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.415160    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-c73f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa96e43d-98b3-4804-bf65-e705013a2e3c-success-200-isvc-c73f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:16:19.574951 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.574922    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"]
May 26 16:16:19.578831 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:19.578805    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c73f1-predictor-9985b79f8-6gnkh"]
May 26 16:16:20.221505 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:20.221468    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:21.235514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:21.235479    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" path="/var/lib/kubelet/pods/aa96e43d-98b3-4804-bf65-e705013a2e3c/volumes"
May 26 16:16:24.258030 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:24.257998    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:16:24.260389 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:24.258471    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:16:25.222039 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:25.222000    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:30.221870 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:30.221833    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:30.222235 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:30.221964    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:34.258714 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:34.258678    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:16:35.222348 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:35.222307    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:40.222192 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:40.222152    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:44.258498 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:44.258459    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:16:45.221848 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.221808    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:45.808848 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.808815    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"]
May 26 16:16:45.809201 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.809191    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kube-rbac-proxy"
May 26 16:16:45.809249 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.809202    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kube-rbac-proxy"
May 26 16:16:45.809249 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.809213    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container"
May 26 16:16:45.809249 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.809220    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container"
May 26 16:16:45.809372 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.809270    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kube-rbac-proxy"
May 26 16:16:45.809372 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.809334    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="aa96e43d-98b3-4804-bf65-e705013a2e3c" containerName="kserve-container"
May 26 16:16:45.812335 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.812319    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:45.814507 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.814489    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
May 26 16:16:45.814628 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.814490    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
May 26 16:16:45.818741 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.818716    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"]
May 26 16:16:45.859997 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.859964    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75381a8-944d-417c-83d5-82de0c255202-proxy-tls\") pod \"model-chainer-5cbf9648f-m44jf\" (UID: \"a75381a8-944d-417c-83d5-82de0c255202\") " pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:45.860165 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.860088    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a75381a8-944d-417c-83d5-82de0c255202-openshift-service-ca-bundle\") pod \"model-chainer-5cbf9648f-m44jf\" (UID: \"a75381a8-944d-417c-83d5-82de0c255202\") " pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:45.961065 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.961027    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75381a8-944d-417c-83d5-82de0c255202-proxy-tls\") pod \"model-chainer-5cbf9648f-m44jf\" (UID: \"a75381a8-944d-417c-83d5-82de0c255202\") " pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:45.961229 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.961106    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a75381a8-944d-417c-83d5-82de0c255202-openshift-service-ca-bundle\") pod \"model-chainer-5cbf9648f-m44jf\" (UID: \"a75381a8-944d-417c-83d5-82de0c255202\") " pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:45.961229 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:16:45.961181    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
May 26 16:16:45.961334 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:16:45.961258    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a75381a8-944d-417c-83d5-82de0c255202-proxy-tls podName:a75381a8-944d-417c-83d5-82de0c255202 nodeName:}" failed. No retries permitted until 2026-05-26 16:16:46.46124136 +0000 UTC m=+957.842563364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a75381a8-944d-417c-83d5-82de0c255202-proxy-tls") pod "model-chainer-5cbf9648f-m44jf" (UID: "a75381a8-944d-417c-83d5-82de0c255202") : secret "model-chainer-serving-cert" not found
May 26 16:16:45.961699 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:45.961682    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a75381a8-944d-417c-83d5-82de0c255202-openshift-service-ca-bundle\") pod \"model-chainer-5cbf9648f-m44jf\" (UID: \"a75381a8-944d-417c-83d5-82de0c255202\") " pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:46.144605 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.144580    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:46.264683 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.264653    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e437aae-3787-4015-bd15-6d58f1c88265-openshift-service-ca-bundle\") pod \"0e437aae-3787-4015-bd15-6d58f1c88265\" (UID: \"0e437aae-3787-4015-bd15-6d58f1c88265\") "
May 26 16:16:46.264848 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.264766    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e437aae-3787-4015-bd15-6d58f1c88265-proxy-tls\") pod \"0e437aae-3787-4015-bd15-6d58f1c88265\" (UID: \"0e437aae-3787-4015-bd15-6d58f1c88265\") "
May 26 16:16:46.264995 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.264973    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e437aae-3787-4015-bd15-6d58f1c88265-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0e437aae-3787-4015-bd15-6d58f1c88265" (UID: "0e437aae-3787-4015-bd15-6d58f1c88265"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:16:46.266800 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.266776    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e437aae-3787-4015-bd15-6d58f1c88265-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e437aae-3787-4015-bd15-6d58f1c88265" (UID: "0e437aae-3787-4015-bd15-6d58f1c88265"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:16:46.346802 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.346712    2570 generic.go:358] "Generic (PLEG): container finished" podID="0e437aae-3787-4015-bd15-6d58f1c88265" containerID="6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca" exitCode=0
May 26 16:16:46.346802 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.346781    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"
May 26 16:16:46.346979 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.346798    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" event={"ID":"0e437aae-3787-4015-bd15-6d58f1c88265","Type":"ContainerDied","Data":"6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca"}
May 26 16:16:46.346979 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.346838    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9" event={"ID":"0e437aae-3787-4015-bd15-6d58f1c88265","Type":"ContainerDied","Data":"5fb1654c49653b332b493265ef16816d3206e5600286a58a187886a502651ee4"}
May 26 16:16:46.346979 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.346857    2570 scope.go:117] "RemoveContainer" containerID="6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca"
May 26 16:16:46.355917 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.355896    2570 scope.go:117] "RemoveContainer" containerID="6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca"
May 26 16:16:46.356172 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:16:46.356153    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca\": container with ID starting with 6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca not found: ID does not exist" containerID="6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca"
May 26 16:16:46.356245 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.356184    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca"} err="failed to get container status \"6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca\": rpc error: code = NotFound desc = could not find container \"6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca\": container with ID starting with 6b1f8e80427c60d78a8ee4327a061b1ce4826b6dfc4fb7280ae4eba0131dc6ca not found: ID does not exist"
May 26 16:16:46.366389 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.366310    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e437aae-3787-4015-bd15-6d58f1c88265-openshift-service-ca-bundle\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:16:46.366389 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.366336    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e437aae-3787-4015-bd15-6d58f1c88265-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:16:46.368090 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.368071    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"]
May 26 16:16:46.373533 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.373513    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c73f1-864fbbd8d6-ns5f9"]
May 26 16:16:46.467384 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.467351    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75381a8-944d-417c-83d5-82de0c255202-proxy-tls\") pod \"model-chainer-5cbf9648f-m44jf\" (UID: \"a75381a8-944d-417c-83d5-82de0c255202\") " pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:46.469809 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.469790    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75381a8-944d-417c-83d5-82de0c255202-proxy-tls\") pod \"model-chainer-5cbf9648f-m44jf\" (UID: \"a75381a8-944d-417c-83d5-82de0c255202\") " pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:46.723463 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.723416    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:46.847458 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:46.847433    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"]
May 26 16:16:46.849978 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:16:46.849953    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75381a8_944d_417c_83d5_82de0c255202.slice/crio-97864dcd855ee178744fe692f6ddfc8b5e8b67fa195d63e625b01119facf4b6c WatchSource:0}: Error finding container 97864dcd855ee178744fe692f6ddfc8b5e8b67fa195d63e625b01119facf4b6c: Status 404 returned error can't find the container with id 97864dcd855ee178744fe692f6ddfc8b5e8b67fa195d63e625b01119facf4b6c
May 26 16:16:47.236418 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:47.236383    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" path="/var/lib/kubelet/pods/0e437aae-3787-4015-bd15-6d58f1c88265/volumes"
May 26 16:16:47.352202 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:47.352162    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" event={"ID":"a75381a8-944d-417c-83d5-82de0c255202","Type":"ContainerStarted","Data":"5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a"}
May 26 16:16:47.352427 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:47.352213    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:47.352427 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:47.352225    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" event={"ID":"a75381a8-944d-417c-83d5-82de0c255202","Type":"ContainerStarted","Data":"97864dcd855ee178744fe692f6ddfc8b5e8b67fa195d63e625b01119facf4b6c"}
May 26 16:16:47.367923 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:47.367880    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" podStartSLOduration=2.367867341 podStartE2EDuration="2.367867341s" podCreationTimestamp="2026-05-26 16:16:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:16:47.366078695 +0000 UTC m=+958.747400723" watchObservedRunningTime="2026-05-26 16:16:47.367867341 +0000 UTC m=+958.749189366"
May 26 16:16:53.364667 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:53.364626    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:16:54.258886 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:54.258842    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:16:55.913396 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:55.913361    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"]
May 26 16:16:55.913768 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:55.913628    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer" containerID="cri-o://5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a" gracePeriod=30
May 26 16:16:56.058566 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.058534    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"]
May 26 16:16:56.058990 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.058973    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1"
May 26 16:16:56.059086 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.058992    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1"
May 26 16:16:56.059146 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.059099    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e437aae-3787-4015-bd15-6d58f1c88265" containerName="switch-graph-c73f1"
May 26 16:16:56.063828 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.063807    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.065975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.065940    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ad62d-kube-rbac-proxy-sar-config\""
May 26 16:16:56.066129 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.066093    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ad62d-predictor-serving-cert\""
May 26 16:16:56.072874 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.072851    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"]
May 26 16:16:56.159079 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.159042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce4a1f89-b0f7-46c3-b82a-537682b708c6-success-200-isvc-ad62d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.159249 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.159085    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wl8w\" (UniqueName: \"kubernetes.io/projected/ce4a1f89-b0f7-46c3-b82a-537682b708c6-kube-api-access-5wl8w\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.159249 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.159126    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce4a1f89-b0f7-46c3-b82a-537682b708c6-proxy-tls\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.259625 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.259535    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce4a1f89-b0f7-46c3-b82a-537682b708c6-success-200-isvc-ad62d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.259625 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.259573    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5wl8w\" (UniqueName: \"kubernetes.io/projected/ce4a1f89-b0f7-46c3-b82a-537682b708c6-kube-api-access-5wl8w\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.259625 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.259607    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce4a1f89-b0f7-46c3-b82a-537682b708c6-proxy-tls\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.259897 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:16:56.259745    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-serving-cert: secret "success-200-isvc-ad62d-predictor-serving-cert" not found
May 26 16:16:56.259897 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:16:56.259802    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce4a1f89-b0f7-46c3-b82a-537682b708c6-proxy-tls podName:ce4a1f89-b0f7-46c3-b82a-537682b708c6 nodeName:}" failed. No retries permitted until 2026-05-26 16:16:56.759787835 +0000 UTC m=+968.141109838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ce4a1f89-b0f7-46c3-b82a-537682b708c6-proxy-tls") pod "success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" (UID: "ce4a1f89-b0f7-46c3-b82a-537682b708c6") : secret "success-200-isvc-ad62d-predictor-serving-cert" not found
May 26 16:16:56.260247 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.260220    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce4a1f89-b0f7-46c3-b82a-537682b708c6-success-200-isvc-ad62d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.270330 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.270271    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wl8w\" (UniqueName: \"kubernetes.io/projected/ce4a1f89-b0f7-46c3-b82a-537682b708c6-kube-api-access-5wl8w\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.763847 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.763806    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce4a1f89-b0f7-46c3-b82a-537682b708c6-proxy-tls\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.766304 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.766248    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce4a1f89-b0f7-46c3-b82a-537682b708c6-proxy-tls\") pod \"success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") " pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:56.974722 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:56.974685    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:57.102371 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:57.102346    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"]
May 26 16:16:57.104268 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:16:57.104240    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4a1f89_b0f7_46c3_b82a_537682b708c6.slice/crio-929a8d2a62e0351f33fa54a4ea9eb33cf97a6626d926ed6038f9c2e731d4d13b WatchSource:0}: Error finding container 929a8d2a62e0351f33fa54a4ea9eb33cf97a6626d926ed6038f9c2e731d4d13b: Status 404 returned error can't find the container with id 929a8d2a62e0351f33fa54a4ea9eb33cf97a6626d926ed6038f9c2e731d4d13b
May 26 16:16:57.391009 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:57.390972    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" event={"ID":"ce4a1f89-b0f7-46c3-b82a-537682b708c6","Type":"ContainerStarted","Data":"06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10"}
May 26 16:16:57.391178 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:57.391017    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" event={"ID":"ce4a1f89-b0f7-46c3-b82a-537682b708c6","Type":"ContainerStarted","Data":"192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798"}
May 26 16:16:57.391178 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:57.391032    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" event={"ID":"ce4a1f89-b0f7-46c3-b82a-537682b708c6","Type":"ContainerStarted","Data":"929a8d2a62e0351f33fa54a4ea9eb33cf97a6626d926ed6038f9c2e731d4d13b"}
May 26 16:16:57.391178 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:57.391147    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:57.409719 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:57.409660    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" podStartSLOduration=1.40964359 podStartE2EDuration="1.40964359s" podCreationTimestamp="2026-05-26 16:16:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:16:57.408556042 +0000 UTC m=+968.789878082" watchObservedRunningTime="2026-05-26 16:16:57.40964359 +0000 UTC m=+968.790965613"
May 26 16:16:58.359994 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:58.359955    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:58.395248 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:58.395215    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:16:58.396722 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:58.396686    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 16:16:59.397707 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:16:59.397668    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 16:17:03.359689 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:03.359649    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:17:04.259155 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:04.259118    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:17:04.401970 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:04.401944    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:17:04.402578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:04.402551    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 16:17:08.360701 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:08.360663    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:17:08.361057 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:08.360771    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:17:13.359951 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:13.359910    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:17:14.402493 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:14.402455    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 16:17:16.238946 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.238914    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"]
May 26 16:17:16.242437 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.242420    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:16.244550 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.244532    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-df0d2-serving-cert\""
May 26 16:17:16.244639 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.244536    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-df0d2-kube-rbac-proxy-sar-config\""
May 26 16:17:16.251997 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.251974    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"]
May 26 16:17:16.339469 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.339436    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da714d88-e1c9-4f61-86e4-e1f154e97f88-proxy-tls\") pod \"switch-graph-df0d2-58df6d94d7-fwb92\" (UID: \"da714d88-e1c9-4f61-86e4-e1f154e97f88\") " pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:16.339618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.339484    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da714d88-e1c9-4f61-86e4-e1f154e97f88-openshift-service-ca-bundle\") pod \"switch-graph-df0d2-58df6d94d7-fwb92\" (UID: \"da714d88-e1c9-4f61-86e4-e1f154e97f88\") " pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:16.440521 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.440484    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da714d88-e1c9-4f61-86e4-e1f154e97f88-proxy-tls\") pod \"switch-graph-df0d2-58df6d94d7-fwb92\" (UID: \"da714d88-e1c9-4f61-86e4-e1f154e97f88\") " pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:16.440702 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.440532    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da714d88-e1c9-4f61-86e4-e1f154e97f88-openshift-service-ca-bundle\") pod \"switch-graph-df0d2-58df6d94d7-fwb92\" (UID: \"da714d88-e1c9-4f61-86e4-e1f154e97f88\") " pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:16.440702 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:17:16.440628    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-df0d2-serving-cert: secret "switch-graph-df0d2-serving-cert" not found
May 26 16:17:16.440702 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:17:16.440693    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da714d88-e1c9-4f61-86e4-e1f154e97f88-proxy-tls podName:da714d88-e1c9-4f61-86e4-e1f154e97f88 nodeName:}" failed. No retries permitted until 2026-05-26 16:17:16.940678098 +0000 UTC m=+988.322000101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/da714d88-e1c9-4f61-86e4-e1f154e97f88-proxy-tls") pod "switch-graph-df0d2-58df6d94d7-fwb92" (UID: "da714d88-e1c9-4f61-86e4-e1f154e97f88") : secret "switch-graph-df0d2-serving-cert" not found
May 26 16:17:16.441171 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.441154    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da714d88-e1c9-4f61-86e4-e1f154e97f88-openshift-service-ca-bundle\") pod \"switch-graph-df0d2-58df6d94d7-fwb92\" (UID: \"da714d88-e1c9-4f61-86e4-e1f154e97f88\") " pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:16.944921 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.944874    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da714d88-e1c9-4f61-86e4-e1f154e97f88-proxy-tls\") pod \"switch-graph-df0d2-58df6d94d7-fwb92\" (UID: \"da714d88-e1c9-4f61-86e4-e1f154e97f88\") " pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:16.947329 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:16.947296    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da714d88-e1c9-4f61-86e4-e1f154e97f88-proxy-tls\") pod \"switch-graph-df0d2-58df6d94d7-fwb92\" (UID: \"da714d88-e1c9-4f61-86e4-e1f154e97f88\") " pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:17.154712 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:17.154661    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:17.288937 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:17.288910    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"]
May 26 16:17:17.291608 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:17:17.291576    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda714d88_e1c9_4f61_86e4_e1f154e97f88.slice/crio-98432ee769a9a2584a6d53612f675e797624fda789920d50cd05cc5c66e28e1c WatchSource:0}: Error finding container 98432ee769a9a2584a6d53612f675e797624fda789920d50cd05cc5c66e28e1c: Status 404 returned error can't find the container with id 98432ee769a9a2584a6d53612f675e797624fda789920d50cd05cc5c66e28e1c
May 26 16:17:17.457986 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:17.457894    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" event={"ID":"da714d88-e1c9-4f61-86e4-e1f154e97f88","Type":"ContainerStarted","Data":"c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4"}
May 26 16:17:17.457986 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:17.457932    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" event={"ID":"da714d88-e1c9-4f61-86e4-e1f154e97f88","Type":"ContainerStarted","Data":"98432ee769a9a2584a6d53612f675e797624fda789920d50cd05cc5c66e28e1c"}
May 26 16:17:17.458210 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:17.458000    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:17.474548 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:17.474503    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" podStartSLOduration=1.474490625 podStartE2EDuration="1.474490625s" podCreationTimestamp="2026-05-26 16:17:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:17:17.472488198 +0000 UTC m=+988.853810220" watchObservedRunningTime="2026-05-26 16:17:17.474490625 +0000 UTC m=+988.855812649"
May 26 16:17:18.359693 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:18.359653    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:17:23.359832 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:23.359793    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:17:23.469433 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:23.469401    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:17:24.402657 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:24.402613    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 16:17:26.067884 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.067859    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:17:26.124831 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.124797    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75381a8-944d-417c-83d5-82de0c255202-proxy-tls\") pod \"a75381a8-944d-417c-83d5-82de0c255202\" (UID: \"a75381a8-944d-417c-83d5-82de0c255202\") "
May 26 16:17:26.124991 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.124875    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a75381a8-944d-417c-83d5-82de0c255202-openshift-service-ca-bundle\") pod \"a75381a8-944d-417c-83d5-82de0c255202\" (UID: \"a75381a8-944d-417c-83d5-82de0c255202\") "
May 26 16:17:26.125184 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.125163    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75381a8-944d-417c-83d5-82de0c255202-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a75381a8-944d-417c-83d5-82de0c255202" (UID: "a75381a8-944d-417c-83d5-82de0c255202"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:17:26.126931 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.126911    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75381a8-944d-417c-83d5-82de0c255202-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a75381a8-944d-417c-83d5-82de0c255202" (UID: "a75381a8-944d-417c-83d5-82de0c255202"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:17:26.226481 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.226399    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a75381a8-944d-417c-83d5-82de0c255202-openshift-service-ca-bundle\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:17:26.226481 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.226433    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a75381a8-944d-417c-83d5-82de0c255202-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:17:26.491825 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.491735    2570 generic.go:358] "Generic (PLEG): container finished" podID="a75381a8-944d-417c-83d5-82de0c255202" containerID="5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a" exitCode=0
May 26 16:17:26.491963 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.491817    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" event={"ID":"a75381a8-944d-417c-83d5-82de0c255202","Type":"ContainerDied","Data":"5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a"}
May 26 16:17:26.491963 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.491853    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf" event={"ID":"a75381a8-944d-417c-83d5-82de0c255202","Type":"ContainerDied","Data":"97864dcd855ee178744fe692f6ddfc8b5e8b67fa195d63e625b01119facf4b6c"}
May 26 16:17:26.491963 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.491868    2570 scope.go:117] "RemoveContainer" containerID="5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a"
May 26 16:17:26.491963 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.491830    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"
May 26 16:17:26.500128 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.500112    2570 scope.go:117] "RemoveContainer" containerID="5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a"
May 26 16:17:26.500399 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:17:26.500378    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a\": container with ID starting with 5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a not found: ID does not exist" containerID="5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a"
May 26 16:17:26.500457 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.500406    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a"} err="failed to get container status \"5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a\": rpc error: code = NotFound desc = could not find container \"5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a\": container with ID starting with 5f63f2428f61790be437c2ee2ce48533a9ecdf133a26d596a6af51a310320e8a not found: ID does not exist"
May 26 16:17:26.512814 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.512788    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"]
May 26 16:17:26.516616 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:26.516594    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5cbf9648f-m44jf"]
May 26 16:17:27.236187 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:27.236148    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75381a8-944d-417c-83d5-82de0c255202" path="/var/lib/kubelet/pods/a75381a8-944d-417c-83d5-82de0c255202/volumes"
May 26 16:17:34.402999 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:34.402959    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 26 16:17:44.403594 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:17:44.403559    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:25:30.920526 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:30.920495    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"]
May 26 16:25:30.921047 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:30.920749    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2" containerID="cri-o://c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4" gracePeriod=30
May 26 16:25:31.065644 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.065610    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"]
May 26 16:25:31.065975 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.065945    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kube-rbac-proxy" containerID="cri-o://288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0" gracePeriod=30
May 26 16:25:31.066076 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.065946    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container" containerID="cri-o://e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05" gracePeriod=30
May 26 16:25:31.184368 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.184329    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"]
May 26 16:25:31.184736 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.184723    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer"
May 26 16:25:31.184778 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.184738    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer"
May 26 16:25:31.184813 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.184797    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="a75381a8-944d-417c-83d5-82de0c255202" containerName="model-chainer"
May 26 16:25:31.187852 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.187836    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.189859 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.189825    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-007a5-kube-rbac-proxy-sar-config\""
May 26 16:25:31.189964 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.189831    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-007a5-predictor-serving-cert\""
May 26 16:25:31.202887 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.202863    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"]
May 26 16:25:31.215184 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.215156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/579e746b-3525-4694-b44a-495beaf07f83-proxy-tls\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.215439 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.215400    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/579e746b-3525-4694-b44a-495beaf07f83-success-200-isvc-007a5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.215580 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.215458    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t4n2\" (UniqueName: \"kubernetes.io/projected/579e746b-3525-4694-b44a-495beaf07f83-kube-api-access-5t4n2\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.316715 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.316680    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/579e746b-3525-4694-b44a-495beaf07f83-success-200-isvc-007a5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.316895 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.316718    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5t4n2\" (UniqueName: \"kubernetes.io/projected/579e746b-3525-4694-b44a-495beaf07f83-kube-api-access-5t4n2\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.316895 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.316806    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/579e746b-3525-4694-b44a-495beaf07f83-proxy-tls\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.317008 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:25:31.316942    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-007a5-predictor-serving-cert: secret "success-200-isvc-007a5-predictor-serving-cert" not found
May 26 16:25:31.317060 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:25:31.317013    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579e746b-3525-4694-b44a-495beaf07f83-proxy-tls podName:579e746b-3525-4694-b44a-495beaf07f83 nodeName:}" failed. No retries permitted until 2026-05-26 16:25:31.816991896 +0000 UTC m=+1483.198313904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/579e746b-3525-4694-b44a-495beaf07f83-proxy-tls") pod "success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" (UID: "579e746b-3525-4694-b44a-495beaf07f83") : secret "success-200-isvc-007a5-predictor-serving-cert" not found
May 26 16:25:31.317452 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.317431    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/579e746b-3525-4694-b44a-495beaf07f83-success-200-isvc-007a5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.325427 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.325405    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t4n2\" (UniqueName: \"kubernetes.io/projected/579e746b-3525-4694-b44a-495beaf07f83-kube-api-access-5t4n2\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.822568 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.822530    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/579e746b-3525-4694-b44a-495beaf07f83-proxy-tls\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:31.825058 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:31.825032    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/579e746b-3525-4694-b44a-495beaf07f83-proxy-tls\") pod \"success-200-isvc-007a5-predictor-6b94cd4f88-tngw8\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") " pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:32.098410 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:32.098315    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:32.155773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:32.155693    2570 generic.go:358] "Generic (PLEG): container finished" podID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerID="288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0" exitCode=2
May 26 16:25:32.155914 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:32.155772    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" event={"ID":"a1d29b3a-27a7-4483-9a6a-afe511aebef2","Type":"ContainerDied","Data":"288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0"}
May 26 16:25:32.228443 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:32.228395    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"]
May 26 16:25:32.230956 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:25:32.230926    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579e746b_3525_4694_b44a_495beaf07f83.slice/crio-8e0f974c526c4efd327033dd7129c99c8051279f8ef4930c0e914020a313ba51 WatchSource:0}: Error finding container 8e0f974c526c4efd327033dd7129c99c8051279f8ef4930c0e914020a313ba51: Status 404 returned error can't find the container with id 8e0f974c526c4efd327033dd7129c99c8051279f8ef4930c0e914020a313ba51
May 26 16:25:32.233012 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:32.232994    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:25:33.160666 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:33.160635    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" event={"ID":"579e746b-3525-4694-b44a-495beaf07f83","Type":"ContainerStarted","Data":"7be36afa27e05bbbf6460d22c043566ccfdede00b484975cf3ac49bcbd2f4821"}
May 26 16:25:33.160666 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:33.160669    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" event={"ID":"579e746b-3525-4694-b44a-495beaf07f83","Type":"ContainerStarted","Data":"58e461c93c20d31c2b684bdddc118b3e88d9e4bca4acd65a4a3258077fa49270"}
May 26 16:25:33.161076 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:33.160700    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:33.161076 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:33.160721    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:33.161076 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:33.160731    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" event={"ID":"579e746b-3525-4694-b44a-495beaf07f83","Type":"ContainerStarted","Data":"8e0f974c526c4efd327033dd7129c99c8051279f8ef4930c0e914020a313ba51"}
May 26 16:25:33.161898 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:33.161877    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 16:25:33.178878 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:33.178836    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podStartSLOduration=2.178823442 podStartE2EDuration="2.178823442s" podCreationTimestamp="2026-05-26 16:25:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:25:33.176137103 +0000 UTC m=+1484.557459129" watchObservedRunningTime="2026-05-26 16:25:33.178823442 +0000 UTC m=+1484.560145463"
May 26 16:25:33.467902 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:33.467815    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:25:34.164354 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.164300    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 16:25:34.320557 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.320532    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:25:34.448485 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.448392    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1d29b3a-27a7-4483-9a6a-afe511aebef2-success-200-isvc-df0d2-kube-rbac-proxy-sar-config\") pod \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") "
May 26 16:25:34.448651 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.448524    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qd9z\" (UniqueName: \"kubernetes.io/projected/a1d29b3a-27a7-4483-9a6a-afe511aebef2-kube-api-access-8qd9z\") pod \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") "
May 26 16:25:34.448651 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.448562    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1d29b3a-27a7-4483-9a6a-afe511aebef2-proxy-tls\") pod \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\" (UID: \"a1d29b3a-27a7-4483-9a6a-afe511aebef2\") "
May 26 16:25:34.448780 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.448717    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d29b3a-27a7-4483-9a6a-afe511aebef2-success-200-isvc-df0d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-df0d2-kube-rbac-proxy-sar-config") pod "a1d29b3a-27a7-4483-9a6a-afe511aebef2" (UID: "a1d29b3a-27a7-4483-9a6a-afe511aebef2"). InnerVolumeSpecName "success-200-isvc-df0d2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:25:34.448885 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.448864    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-df0d2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1d29b3a-27a7-4483-9a6a-afe511aebef2-success-200-isvc-df0d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:25:34.450542 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.450515    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d29b3a-27a7-4483-9a6a-afe511aebef2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1d29b3a-27a7-4483-9a6a-afe511aebef2" (UID: "a1d29b3a-27a7-4483-9a6a-afe511aebef2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:25:34.450675 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.450611    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d29b3a-27a7-4483-9a6a-afe511aebef2-kube-api-access-8qd9z" (OuterVolumeSpecName: "kube-api-access-8qd9z") pod "a1d29b3a-27a7-4483-9a6a-afe511aebef2" (UID: "a1d29b3a-27a7-4483-9a6a-afe511aebef2"). InnerVolumeSpecName "kube-api-access-8qd9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:25:34.549758 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.549710    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qd9z\" (UniqueName: \"kubernetes.io/projected/a1d29b3a-27a7-4483-9a6a-afe511aebef2-kube-api-access-8qd9z\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:25:34.549758 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:34.549745    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1d29b3a-27a7-4483-9a6a-afe511aebef2-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:25:35.168639 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.168601    2570 generic.go:358] "Generic (PLEG): container finished" podID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerID="e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05" exitCode=0
May 26 16:25:35.169034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.168675    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"
May 26 16:25:35.169034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.168690    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" event={"ID":"a1d29b3a-27a7-4483-9a6a-afe511aebef2","Type":"ContainerDied","Data":"e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05"}
May 26 16:25:35.169034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.168732    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" event={"ID":"a1d29b3a-27a7-4483-9a6a-afe511aebef2","Type":"ContainerDied","Data":"de762552845345bd628eeb4a4feeee0d0eccd0a7350012d2d5e9e250bdbf8c24"}
May 26 16:25:35.169034 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.168748    2570 scope.go:117] "RemoveContainer" containerID="288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0"
May 26 16:25:35.177160 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.177143    2570 scope.go:117] "RemoveContainer" containerID="e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05"
May 26 16:25:35.184501 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.184349    2570 scope.go:117] "RemoveContainer" containerID="288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0"
May 26 16:25:35.184620 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:25:35.184599    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0\": container with ID starting with 288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0 not found: ID does not exist" containerID="288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0"
May 26 16:25:35.184686 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.184632    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0"} err="failed to get container status \"288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0\": rpc error: code = NotFound desc = could not find container \"288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0\": container with ID starting with 288593bc053d73f099c7da4cfaa4820c6208663068dcebe2f216d63157e9dfb0 not found: ID does not exist"
May 26 16:25:35.184686 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.184657    2570 scope.go:117] "RemoveContainer" containerID="e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05"
May 26 16:25:35.184886 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:25:35.184868    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05\": container with ID starting with e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05 not found: ID does not exist" containerID="e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05"
May 26 16:25:35.184925 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.184891    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05"} err="failed to get container status \"e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05\": rpc error: code = NotFound desc = could not find container \"e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05\": container with ID starting with e0909f578e4140feb1159d62382fbfccc7bef8c124cb944f0a2f617469795e05 not found: ID does not exist"
May 26 16:25:35.193503 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.193477    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"]
May 26 16:25:35.197168 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.197147    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq"]
May 26 16:25:35.236055 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.236025    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" path="/var/lib/kubelet/pods/a1d29b3a-27a7-4483-9a6a-afe511aebef2/volumes"
May 26 16:25:35.254446 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.254402    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": context deadline exceeded"
May 26 16:25:35.259713 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:35.259688    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df0d2-predictor-8fddf9f46-js5fq" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: i/o timeout"
May 26 16:25:38.468494 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:38.468457    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:25:39.168535 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:39.168503    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:25:39.168996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:39.168973    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 16:25:43.468407 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:43.468366    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:25:43.468785 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:43.468471    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:25:48.468430 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:48.468388    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:25:49.168932 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:49.168892    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 16:25:53.468027 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:53.467986    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:25:58.468505 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:58.468467    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:25:59.169465 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:25:59.169428    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 16:26:01.064657 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.064632    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:26:01.183110 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.182969    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da714d88-e1c9-4f61-86e4-e1f154e97f88-openshift-service-ca-bundle\") pod \"da714d88-e1c9-4f61-86e4-e1f154e97f88\" (UID: \"da714d88-e1c9-4f61-86e4-e1f154e97f88\") "
May 26 16:26:01.183261 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.183136    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da714d88-e1c9-4f61-86e4-e1f154e97f88-proxy-tls\") pod \"da714d88-e1c9-4f61-86e4-e1f154e97f88\" (UID: \"da714d88-e1c9-4f61-86e4-e1f154e97f88\") "
May 26 16:26:01.183369 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.183256    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da714d88-e1c9-4f61-86e4-e1f154e97f88-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "da714d88-e1c9-4f61-86e4-e1f154e97f88" (UID: "da714d88-e1c9-4f61-86e4-e1f154e97f88"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:26:01.183429 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.183390    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da714d88-e1c9-4f61-86e4-e1f154e97f88-openshift-service-ca-bundle\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:26:01.185237 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.185206    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da714d88-e1c9-4f61-86e4-e1f154e97f88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da714d88-e1c9-4f61-86e4-e1f154e97f88" (UID: "da714d88-e1c9-4f61-86e4-e1f154e97f88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:26:01.259501 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.259458    2570 generic.go:358] "Generic (PLEG): container finished" podID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerID="c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4" exitCode=0
May 26 16:26:01.259647 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.259525    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"
May 26 16:26:01.259647 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.259534    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" event={"ID":"da714d88-e1c9-4f61-86e4-e1f154e97f88","Type":"ContainerDied","Data":"c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4"}
May 26 16:26:01.259647 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.259574    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92" event={"ID":"da714d88-e1c9-4f61-86e4-e1f154e97f88","Type":"ContainerDied","Data":"98432ee769a9a2584a6d53612f675e797624fda789920d50cd05cc5c66e28e1c"}
May 26 16:26:01.259647 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.259589    2570 scope.go:117] "RemoveContainer" containerID="c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4"
May 26 16:26:01.268144 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.268127    2570 scope.go:117] "RemoveContainer" containerID="c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4"
May 26 16:26:01.268494 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:26:01.268470    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4\": container with ID starting with c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4 not found: ID does not exist" containerID="c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4"
May 26 16:26:01.268634 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.268496    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4"} err="failed to get container status \"c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4\": rpc error: code = NotFound desc = could not find container \"c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4\": container with ID starting with c71110f1df49a40df76dec555a1fe6c3c626f5d37b94ddcb90f02beb763bbfb4 not found: ID does not exist"
May 26 16:26:01.276805 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.276781    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"]
May 26 16:26:01.281895 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.281870    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-df0d2-58df6d94d7-fwb92"]
May 26 16:26:01.284228 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:01.284205    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da714d88-e1c9-4f61-86e4-e1f154e97f88-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:26:03.236224 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:03.236191    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" path="/var/lib/kubelet/pods/da714d88-e1c9-4f61-86e4-e1f154e97f88/volumes"
May 26 16:26:09.169508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:09.169426    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
May 26 16:26:10.917003 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.916964    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"]
May 26 16:26:10.917514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.917376    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container" containerID="cri-o://192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798" gracePeriod=30
May 26 16:26:10.917514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.917423    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kube-rbac-proxy" containerID="cri-o://06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10" gracePeriod=30
May 26 16:26:10.942640 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.942612    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"]
May 26 16:26:10.942973 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.942961    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2"
May 26 16:26:10.943018 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.942977    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2"
May 26 16:26:10.943018 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.943004    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kube-rbac-proxy"
May 26 16:26:10.943018 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.943013    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kube-rbac-proxy"
May 26 16:26:10.943127 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.943037    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container"
May 26 16:26:10.943127 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.943045    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container"
May 26 16:26:10.943127 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.943119    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kserve-container"
May 26 16:26:10.943247 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.943133    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="a1d29b3a-27a7-4483-9a6a-afe511aebef2" containerName="kube-rbac-proxy"
May 26 16:26:10.943247 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.943146    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="da714d88-e1c9-4f61-86e4-e1f154e97f88" containerName="switch-graph-df0d2"
May 26 16:26:10.947773 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.947757    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:10.949890 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.949866    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a007a-kube-rbac-proxy-sar-config\""
May 26 16:26:10.949984 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.949928    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a007a-predictor-serving-cert\""
May 26 16:26:10.957383 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.957360    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"]
May 26 16:26:10.962424 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.962401    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168729a0-3ea9-4f06-a196-fec2df57a92c-success-200-isvc-a007a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:10.962525 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.962436    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168729a0-3ea9-4f06-a196-fec2df57a92c-proxy-tls\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:10.962525 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:10.962454    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49s6x\" (UniqueName: \"kubernetes.io/projected/168729a0-3ea9-4f06-a196-fec2df57a92c-kube-api-access-49s6x\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:11.063535 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.063499    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168729a0-3ea9-4f06-a196-fec2df57a92c-proxy-tls\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:11.063879 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.063544    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-49s6x\" (UniqueName: \"kubernetes.io/projected/168729a0-3ea9-4f06-a196-fec2df57a92c-kube-api-access-49s6x\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:11.063879 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:26:11.063663    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a007a-predictor-serving-cert: secret "success-200-isvc-a007a-predictor-serving-cert" not found
May 26 16:26:11.063879 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.063696    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168729a0-3ea9-4f06-a196-fec2df57a92c-success-200-isvc-a007a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:11.063879 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:26:11.063734    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/168729a0-3ea9-4f06-a196-fec2df57a92c-proxy-tls podName:168729a0-3ea9-4f06-a196-fec2df57a92c nodeName:}" failed. No retries permitted until 2026-05-26 16:26:11.563711263 +0000 UTC m=+1522.945033267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/168729a0-3ea9-4f06-a196-fec2df57a92c-proxy-tls") pod "success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" (UID: "168729a0-3ea9-4f06-a196-fec2df57a92c") : secret "success-200-isvc-a007a-predictor-serving-cert" not found
May 26 16:26:11.064258 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.064239    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168729a0-3ea9-4f06-a196-fec2df57a92c-success-200-isvc-a007a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:11.071819 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.071792    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49s6x\" (UniqueName: \"kubernetes.io/projected/168729a0-3ea9-4f06-a196-fec2df57a92c-kube-api-access-49s6x\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:11.294738 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.294654    2570 generic.go:358] "Generic (PLEG): container finished" podID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerID="06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10" exitCode=2
May 26 16:26:11.294738 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.294726    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" event={"ID":"ce4a1f89-b0f7-46c3-b82a-537682b708c6","Type":"ContainerDied","Data":"06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10"}
May 26 16:26:11.567541 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.567447    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168729a0-3ea9-4f06-a196-fec2df57a92c-proxy-tls\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:11.569767 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.569734    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168729a0-3ea9-4f06-a196-fec2df57a92c-proxy-tls\") pod \"success-200-isvc-a007a-predictor-5f5fc59776-qcnp4\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") " pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:11.859381 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.859268    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:11.990098 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:11.990070    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"]
May 26 16:26:11.994418 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:26:11.994388    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168729a0_3ea9_4f06_a196_fec2df57a92c.slice/crio-f2906564188f684ab8a4715f2c8640372e785f21411a61a6ff4cb4ffa67ef94a WatchSource:0}: Error finding container f2906564188f684ab8a4715f2c8640372e785f21411a61a6ff4cb4ffa67ef94a: Status 404 returned error can't find the container with id f2906564188f684ab8a4715f2c8640372e785f21411a61a6ff4cb4ffa67ef94a
May 26 16:26:12.300238 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:12.300197    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" event={"ID":"168729a0-3ea9-4f06-a196-fec2df57a92c","Type":"ContainerStarted","Data":"1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76"}
May 26 16:26:12.300238 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:12.300241    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" event={"ID":"168729a0-3ea9-4f06-a196-fec2df57a92c","Type":"ContainerStarted","Data":"d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6"}
May 26 16:26:12.300464 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:12.300253    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" event={"ID":"168729a0-3ea9-4f06-a196-fec2df57a92c","Type":"ContainerStarted","Data":"f2906564188f684ab8a4715f2c8640372e785f21411a61a6ff4cb4ffa67ef94a"}
May 26 16:26:12.300464 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:12.300317    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:12.318367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:12.318320    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podStartSLOduration=2.318306008 podStartE2EDuration="2.318306008s" podCreationTimestamp="2026-05-26 16:26:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:26:12.316218962 +0000 UTC m=+1523.697541009" watchObservedRunningTime="2026-05-26 16:26:12.318306008 +0000 UTC m=+1523.699628034"
May 26 16:26:13.303826 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:13.303789    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:13.305042 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:13.305017    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 16:26:14.175857 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.175835    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:26:14.188219 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.188196    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce4a1f89-b0f7-46c3-b82a-537682b708c6-proxy-tls\") pod \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") "
May 26 16:26:14.188393 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.188302    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wl8w\" (UniqueName: \"kubernetes.io/projected/ce4a1f89-b0f7-46c3-b82a-537682b708c6-kube-api-access-5wl8w\") pod \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") "
May 26 16:26:14.188393 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.188342    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce4a1f89-b0f7-46c3-b82a-537682b708c6-success-200-isvc-ad62d-kube-rbac-proxy-sar-config\") pod \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\" (UID: \"ce4a1f89-b0f7-46c3-b82a-537682b708c6\") "
May 26 16:26:14.188700 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.188676    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4a1f89-b0f7-46c3-b82a-537682b708c6-success-200-isvc-ad62d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-ad62d-kube-rbac-proxy-sar-config") pod "ce4a1f89-b0f7-46c3-b82a-537682b708c6" (UID: "ce4a1f89-b0f7-46c3-b82a-537682b708c6"). InnerVolumeSpecName "success-200-isvc-ad62d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:26:14.190184 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.190161    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4a1f89-b0f7-46c3-b82a-537682b708c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce4a1f89-b0f7-46c3-b82a-537682b708c6" (UID: "ce4a1f89-b0f7-46c3-b82a-537682b708c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:26:14.190288 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.190248    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4a1f89-b0f7-46c3-b82a-537682b708c6-kube-api-access-5wl8w" (OuterVolumeSpecName: "kube-api-access-5wl8w") pod "ce4a1f89-b0f7-46c3-b82a-537682b708c6" (UID: "ce4a1f89-b0f7-46c3-b82a-537682b708c6"). InnerVolumeSpecName "kube-api-access-5wl8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:26:14.289869 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.289835    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce4a1f89-b0f7-46c3-b82a-537682b708c6-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:26:14.289869 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.289862    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wl8w\" (UniqueName: \"kubernetes.io/projected/ce4a1f89-b0f7-46c3-b82a-537682b708c6-kube-api-access-5wl8w\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:26:14.289869 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.289873    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-ad62d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce4a1f89-b0f7-46c3-b82a-537682b708c6-success-200-isvc-ad62d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:26:14.307651 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.307618    2570 generic.go:358] "Generic (PLEG): container finished" podID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerID="192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798" exitCode=0
May 26 16:26:14.308036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.307684    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"
May 26 16:26:14.308036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.307702    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" event={"ID":"ce4a1f89-b0f7-46c3-b82a-537682b708c6","Type":"ContainerDied","Data":"192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798"}
May 26 16:26:14.308036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.307738    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68" event={"ID":"ce4a1f89-b0f7-46c3-b82a-537682b708c6","Type":"ContainerDied","Data":"929a8d2a62e0351f33fa54a4ea9eb33cf97a6626d926ed6038f9c2e731d4d13b"}
May 26 16:26:14.308036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.307754    2570 scope.go:117] "RemoveContainer" containerID="06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10"
May 26 16:26:14.308324 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.308294    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 16:26:14.316244 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.316228    2570 scope.go:117] "RemoveContainer" containerID="192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798"
May 26 16:26:14.323567 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.323398    2570 scope.go:117] "RemoveContainer" containerID="06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10"
May 26 16:26:14.323644 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:26:14.323623    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10\": container with ID starting with 06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10 not found: ID does not exist" containerID="06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10"
May 26 16:26:14.323696 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.323653    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10"} err="failed to get container status \"06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10\": rpc error: code = NotFound desc = could not find container \"06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10\": container with ID starting with 06921c4cb45baf2e370e0e12999d71c9418054338f6f5587746ce25f02426b10 not found: ID does not exist"
May 26 16:26:14.323696 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.323669    2570 scope.go:117] "RemoveContainer" containerID="192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798"
May 26 16:26:14.323882 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:26:14.323868    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798\": container with ID starting with 192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798 not found: ID does not exist" containerID="192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798"
May 26 16:26:14.323918 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.323886    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798"} err="failed to get container status \"192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798\": rpc error: code = NotFound desc = could not find container \"192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798\": container with ID starting with 192cbd70cbb9b7aa4a7c3431a0490819c6a80486d09cfb41821fbc145af61798 not found: ID does not exist"
May 26 16:26:14.329661 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.329629    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"]
May 26 16:26:14.332449 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:14.332431    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ad62d-predictor-75c98cb7b4-6gj68"]
May 26 16:26:15.236347 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:15.236309    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" path="/var/lib/kubelet/pods/ce4a1f89-b0f7-46c3-b82a-537682b708c6/volumes"
May 26 16:26:19.169454 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:19.169424    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:26:19.312778 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:19.312751    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:19.313358 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:19.313329    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 16:26:29.313940 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:29.313897    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 16:26:39.313800 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:39.313760    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 16:26:41.365217 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.365180    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"]
May 26 16:26:41.365714 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.365605    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kube-rbac-proxy" containerID="cri-o://7be36afa27e05bbbf6460d22c043566ccfdede00b484975cf3ac49bcbd2f4821" gracePeriod=30
May 26 16:26:41.365714 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.365598    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container" containerID="cri-o://58e461c93c20d31c2b684bdddc118b3e88d9e4bca4acd65a4a3258077fa49270" gracePeriod=30
May 26 16:26:41.407036 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.406990    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"]
May 26 16:26:41.407472 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.407454    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kube-rbac-proxy"
May 26 16:26:41.407572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.407474    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kube-rbac-proxy"
May 26 16:26:41.407572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.407492    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container"
May 26 16:26:41.407572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.407502    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container"
May 26 16:26:41.407735 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.407618    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kserve-container"
May 26 16:26:41.407735 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.407631    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce4a1f89-b0f7-46c3-b82a-537682b708c6" containerName="kube-rbac-proxy"
May 26 16:26:41.411256 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.411235    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:41.413563 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.413546    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2138e-predictor-serving-cert\""
May 26 16:26:41.413656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.413567    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2138e-kube-rbac-proxy-sar-config\""
May 26 16:26:41.419452 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.419426    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"]
May 26 16:26:41.533398 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.533367    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx7b7\" (UniqueName: \"kubernetes.io/projected/dcd5a011-9458-4571-8981-038c5a4f73b7-kube-api-access-gx7b7\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:41.533555 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.533431    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5a011-9458-4571-8981-038c5a4f73b7-proxy-tls\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:41.533555 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.533502    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5a011-9458-4571-8981-038c5a4f73b7-success-200-isvc-2138e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:41.634496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.634406    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gx7b7\" (UniqueName: \"kubernetes.io/projected/dcd5a011-9458-4571-8981-038c5a4f73b7-kube-api-access-gx7b7\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:41.634496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.634466    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5a011-9458-4571-8981-038c5a4f73b7-proxy-tls\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:41.634496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.634491    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5a011-9458-4571-8981-038c5a4f73b7-success-200-isvc-2138e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:41.634751 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:26:41.634616    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-2138e-predictor-serving-cert: secret "success-200-isvc-2138e-predictor-serving-cert" not found
May 26 16:26:41.634751 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:26:41.634701    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd5a011-9458-4571-8981-038c5a4f73b7-proxy-tls podName:dcd5a011-9458-4571-8981-038c5a4f73b7 nodeName:}" failed. No retries permitted until 2026-05-26 16:26:42.134678988 +0000 UTC m=+1553.516000996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dcd5a011-9458-4571-8981-038c5a4f73b7-proxy-tls") pod "success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" (UID: "dcd5a011-9458-4571-8981-038c5a4f73b7") : secret "success-200-isvc-2138e-predictor-serving-cert" not found
May 26 16:26:41.635108 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.635090    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5a011-9458-4571-8981-038c5a4f73b7-success-200-isvc-2138e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:41.644859 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:41.644831    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx7b7\" (UniqueName: \"kubernetes.io/projected/dcd5a011-9458-4571-8981-038c5a4f73b7-kube-api-access-gx7b7\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:42.139715 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:42.139682    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5a011-9458-4571-8981-038c5a4f73b7-proxy-tls\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:42.142223 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:42.142187    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5a011-9458-4571-8981-038c5a4f73b7-proxy-tls\") pod \"success-200-isvc-2138e-predictor-5d4757c55b-zwp2f\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") " pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:42.322713 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:42.322673    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:42.411578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:42.411548    2570 generic.go:358] "Generic (PLEG): container finished" podID="579e746b-3525-4694-b44a-495beaf07f83" containerID="7be36afa27e05bbbf6460d22c043566ccfdede00b484975cf3ac49bcbd2f4821" exitCode=2
May 26 16:26:42.411984 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:42.411613    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" event={"ID":"579e746b-3525-4694-b44a-495beaf07f83","Type":"ContainerDied","Data":"7be36afa27e05bbbf6460d22c043566ccfdede00b484975cf3ac49bcbd2f4821"}
May 26 16:26:42.453125 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:42.453084    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"]
May 26 16:26:42.456090 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:26:42.456060    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd5a011_9458_4571_8981_038c5a4f73b7.slice/crio-3ecdb088e8b6631461dfa6671e91e4400e7208c8e29ea38fa3db0c4e92285430 WatchSource:0}: Error finding container 3ecdb088e8b6631461dfa6671e91e4400e7208c8e29ea38fa3db0c4e92285430: Status 404 returned error can't find the container with id 3ecdb088e8b6631461dfa6671e91e4400e7208c8e29ea38fa3db0c4e92285430
May 26 16:26:43.416152 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:43.416115    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" event={"ID":"dcd5a011-9458-4571-8981-038c5a4f73b7","Type":"ContainerStarted","Data":"44b914c3f94b72a335dfb083f6f607f39c0463f6c55383b9ae3263d4e88e2163"}
May 26 16:26:43.416152 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:43.416154    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" event={"ID":"dcd5a011-9458-4571-8981-038c5a4f73b7","Type":"ContainerStarted","Data":"59fc08341bc4e94b614a7004b6db7012d99842f2da2d9695c10238fac56d0790"}
May 26 16:26:43.416697 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:43.416166    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" event={"ID":"dcd5a011-9458-4571-8981-038c5a4f73b7","Type":"ContainerStarted","Data":"3ecdb088e8b6631461dfa6671e91e4400e7208c8e29ea38fa3db0c4e92285430"}
May 26 16:26:43.416697 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:43.416321    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:43.416697 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:43.416354    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:43.417760 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:43.417734    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 16:26:43.434841 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:43.434796    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" podStartSLOduration=2.434784594 podStartE2EDuration="2.434784594s" podCreationTimestamp="2026-05-26 16:26:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:26:43.432111715 +0000 UTC m=+1554.813433762" watchObservedRunningTime="2026-05-26 16:26:43.434784594 +0000 UTC m=+1554.816106618"
May 26 16:26:44.165131 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.165085    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
May 26 16:26:44.421618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.421530    2570 generic.go:358] "Generic (PLEG): container finished" podID="579e746b-3525-4694-b44a-495beaf07f83" containerID="58e461c93c20d31c2b684bdddc118b3e88d9e4bca4acd65a4a3258077fa49270" exitCode=0
May 26 16:26:44.421618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.421595    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" event={"ID":"579e746b-3525-4694-b44a-495beaf07f83","Type":"ContainerDied","Data":"58e461c93c20d31c2b684bdddc118b3e88d9e4bca4acd65a4a3258077fa49270"}
May 26 16:26:44.422011 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.421991    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 16:26:44.521361 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.521338    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:26:44.665189 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.665156    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/579e746b-3525-4694-b44a-495beaf07f83-proxy-tls\") pod \"579e746b-3525-4694-b44a-495beaf07f83\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") "
May 26 16:26:44.665390 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.665257    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/579e746b-3525-4694-b44a-495beaf07f83-success-200-isvc-007a5-kube-rbac-proxy-sar-config\") pod \"579e746b-3525-4694-b44a-495beaf07f83\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") "
May 26 16:26:44.665390 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.665329    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t4n2\" (UniqueName: \"kubernetes.io/projected/579e746b-3525-4694-b44a-495beaf07f83-kube-api-access-5t4n2\") pod \"579e746b-3525-4694-b44a-495beaf07f83\" (UID: \"579e746b-3525-4694-b44a-495beaf07f83\") "
May 26 16:26:44.665619 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.665596    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579e746b-3525-4694-b44a-495beaf07f83-success-200-isvc-007a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-007a5-kube-rbac-proxy-sar-config") pod "579e746b-3525-4694-b44a-495beaf07f83" (UID: "579e746b-3525-4694-b44a-495beaf07f83"). InnerVolumeSpecName "success-200-isvc-007a5-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:26:44.667287 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.667243    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579e746b-3525-4694-b44a-495beaf07f83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "579e746b-3525-4694-b44a-495beaf07f83" (UID: "579e746b-3525-4694-b44a-495beaf07f83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:26:44.667287 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.667263    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579e746b-3525-4694-b44a-495beaf07f83-kube-api-access-5t4n2" (OuterVolumeSpecName: "kube-api-access-5t4n2") pod "579e746b-3525-4694-b44a-495beaf07f83" (UID: "579e746b-3525-4694-b44a-495beaf07f83"). InnerVolumeSpecName "kube-api-access-5t4n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:26:44.766572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.766536    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/579e746b-3525-4694-b44a-495beaf07f83-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:26:44.766572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.766567    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-007a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/579e746b-3525-4694-b44a-495beaf07f83-success-200-isvc-007a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:26:44.766572 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:44.766577    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t4n2\" (UniqueName: \"kubernetes.io/projected/579e746b-3525-4694-b44a-495beaf07f83-kube-api-access-5t4n2\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:26:45.426328 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:45.426269    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8" event={"ID":"579e746b-3525-4694-b44a-495beaf07f83","Type":"ContainerDied","Data":"8e0f974c526c4efd327033dd7129c99c8051279f8ef4930c0e914020a313ba51"}
May 26 16:26:45.426720 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:45.426346    2570 scope.go:117] "RemoveContainer" containerID="7be36afa27e05bbbf6460d22c043566ccfdede00b484975cf3ac49bcbd2f4821"
May 26 16:26:45.426720 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:45.426352    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"
May 26 16:26:45.434495 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:45.434475    2570 scope.go:117] "RemoveContainer" containerID="58e461c93c20d31c2b684bdddc118b3e88d9e4bca4acd65a4a3258077fa49270"
May 26 16:26:45.446674 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:45.446648    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"]
May 26 16:26:45.449968 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:45.449944    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-007a5-predictor-6b94cd4f88-tngw8"]
May 26 16:26:47.236448 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:47.236420    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="579e746b-3525-4694-b44a-495beaf07f83" path="/var/lib/kubelet/pods/579e746b-3525-4694-b44a-495beaf07f83/volumes"
May 26 16:26:49.313824 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:49.313795    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
May 26 16:26:49.427173 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:49.427146    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:26:49.427629 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:49.427599    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 16:26:59.314112 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:59.314081    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:26:59.427704 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:26:59.427668    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 16:27:09.428256 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:09.428215    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 16:27:19.427969 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:19.427928    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
May 26 16:27:21.137757 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.137716    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"]
May 26 16:27:21.138127 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.138069    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container" containerID="cri-o://d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6" gracePeriod=30
May 26 16:27:21.138185 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.138099    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kube-rbac-proxy" containerID="cri-o://1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76" gracePeriod=30
May 26 16:27:21.172684 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.172647    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"]
May 26 16:27:21.173081 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.173068    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container"
May 26 16:27:21.173127 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.173083    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container"
May 26 16:27:21.173127 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.173091    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kube-rbac-proxy"
May 26 16:27:21.173127 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.173096    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kube-rbac-proxy"
May 26 16:27:21.173225 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.173172    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kube-rbac-proxy"
May 26 16:27:21.173225 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.173184    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="579e746b-3525-4694-b44a-495beaf07f83" containerName="kserve-container"
May 26 16:27:21.176579 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.176560    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.178656 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.178635    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-86373-predictor-serving-cert\""
May 26 16:27:21.178759 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.178740    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-86373-kube-rbac-proxy-sar-config\""
May 26 16:27:21.186372 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.186338    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"]
May 26 16:27:21.295114 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.295081    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-proxy-tls\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.295298 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.295182    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmq2\" (UniqueName: \"kubernetes.io/projected/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-kube-api-access-4cmq2\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.295298 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.295237    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-success-200-isvc-86373-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.396560 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.396477    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmq2\" (UniqueName: \"kubernetes.io/projected/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-kube-api-access-4cmq2\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.396560 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.396529    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-success-200-isvc-86373-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.396781 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.396567    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-proxy-tls\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.396781 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:27:21.396666    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-86373-predictor-serving-cert: secret "success-200-isvc-86373-predictor-serving-cert" not found
May 26 16:27:21.396781 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:27:21.396751    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-proxy-tls podName:fffa82fa-f5e1-4b1f-a032-1145a6df65f4 nodeName:}" failed. No retries permitted until 2026-05-26 16:27:21.896732951 +0000 UTC m=+1593.278054953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-proxy-tls") pod "success-200-isvc-86373-predictor-59f94db984-rq2zd" (UID: "fffa82fa-f5e1-4b1f-a032-1145a6df65f4") : secret "success-200-isvc-86373-predictor-serving-cert" not found
May 26 16:27:21.397187 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.397163    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-success-200-isvc-86373-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.405890 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.405852    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmq2\" (UniqueName: \"kubernetes.io/projected/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-kube-api-access-4cmq2\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.553115 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.553074    2570 generic.go:358] "Generic (PLEG): container finished" podID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerID="1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76" exitCode=2
May 26 16:27:21.553273 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.553147    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" event={"ID":"168729a0-3ea9-4f06-a196-fec2df57a92c","Type":"ContainerDied","Data":"1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76"}
May 26 16:27:21.901196 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.901157    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-proxy-tls\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:21.903531 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:21.903504    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-proxy-tls\") pod \"success-200-isvc-86373-predictor-59f94db984-rq2zd\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") " pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:22.089728 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:22.089692    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:22.223805 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:22.223782    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"]
May 26 16:27:22.226127 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:27:22.226096    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffa82fa_f5e1_4b1f_a032_1145a6df65f4.slice/crio-0711b170ada4181c5cdddbd572aef98b2e24be857dea414f05369d966a02baa9 WatchSource:0}: Error finding container 0711b170ada4181c5cdddbd572aef98b2e24be857dea414f05369d966a02baa9: Status 404 returned error can't find the container with id 0711b170ada4181c5cdddbd572aef98b2e24be857dea414f05369d966a02baa9
May 26 16:27:22.558965 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:22.558880    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" event={"ID":"fffa82fa-f5e1-4b1f-a032-1145a6df65f4","Type":"ContainerStarted","Data":"574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805"}
May 26 16:27:22.558965 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:22.558914    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" event={"ID":"fffa82fa-f5e1-4b1f-a032-1145a6df65f4","Type":"ContainerStarted","Data":"68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322"}
May 26 16:27:22.558965 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:22.558927    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" event={"ID":"fffa82fa-f5e1-4b1f-a032-1145a6df65f4","Type":"ContainerStarted","Data":"0711b170ada4181c5cdddbd572aef98b2e24be857dea414f05369d966a02baa9"}
May 26 16:27:22.559184 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:22.559027    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:22.559184 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:22.559057    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:22.560315 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:22.560272    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 16:27:22.577437 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:22.577395    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podStartSLOduration=1.577383519 podStartE2EDuration="1.577383519s" podCreationTimestamp="2026-05-26 16:27:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:27:22.575502818 +0000 UTC m=+1593.956824842" watchObservedRunningTime="2026-05-26 16:27:22.577383519 +0000 UTC m=+1593.958705550"
May 26 16:27:23.562496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:23.562459    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 16:27:24.390955 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.390930    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:27:24.524903 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.524820    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168729a0-3ea9-4f06-a196-fec2df57a92c-proxy-tls\") pod \"168729a0-3ea9-4f06-a196-fec2df57a92c\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") "
May 26 16:27:24.524903 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.524860    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168729a0-3ea9-4f06-a196-fec2df57a92c-success-200-isvc-a007a-kube-rbac-proxy-sar-config\") pod \"168729a0-3ea9-4f06-a196-fec2df57a92c\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") "
May 26 16:27:24.525137 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.524946    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49s6x\" (UniqueName: \"kubernetes.io/projected/168729a0-3ea9-4f06-a196-fec2df57a92c-kube-api-access-49s6x\") pod \"168729a0-3ea9-4f06-a196-fec2df57a92c\" (UID: \"168729a0-3ea9-4f06-a196-fec2df57a92c\") "
May 26 16:27:24.525309 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.525257    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168729a0-3ea9-4f06-a196-fec2df57a92c-success-200-isvc-a007a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-a007a-kube-rbac-proxy-sar-config") pod "168729a0-3ea9-4f06-a196-fec2df57a92c" (UID: "168729a0-3ea9-4f06-a196-fec2df57a92c"). InnerVolumeSpecName "success-200-isvc-a007a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:27:24.526873 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.526847    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168729a0-3ea9-4f06-a196-fec2df57a92c-kube-api-access-49s6x" (OuterVolumeSpecName: "kube-api-access-49s6x") pod "168729a0-3ea9-4f06-a196-fec2df57a92c" (UID: "168729a0-3ea9-4f06-a196-fec2df57a92c"). InnerVolumeSpecName "kube-api-access-49s6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:27:24.526930 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.526851    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168729a0-3ea9-4f06-a196-fec2df57a92c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "168729a0-3ea9-4f06-a196-fec2df57a92c" (UID: "168729a0-3ea9-4f06-a196-fec2df57a92c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:27:24.566949 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.566918    2570 generic.go:358] "Generic (PLEG): container finished" podID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerID="d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6" exitCode=0
May 26 16:27:24.567332 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.566993    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"
May 26 16:27:24.567332 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.567000    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" event={"ID":"168729a0-3ea9-4f06-a196-fec2df57a92c","Type":"ContainerDied","Data":"d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6"}
May 26 16:27:24.567332 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.567040    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" event={"ID":"168729a0-3ea9-4f06-a196-fec2df57a92c","Type":"ContainerDied","Data":"f2906564188f684ab8a4715f2c8640372e785f21411a61a6ff4cb4ffa67ef94a"}
May 26 16:27:24.567332 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.567055    2570 scope.go:117] "RemoveContainer" containerID="1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76"
May 26 16:27:24.575347 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.575330    2570 scope.go:117] "RemoveContainer" containerID="d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6"
May 26 16:27:24.582765 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.582336    2570 scope.go:117] "RemoveContainer" containerID="1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76"
May 26 16:27:24.582765 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:27:24.582575    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76\": container with ID starting with 1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76 not found: ID does not exist" containerID="1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76"
May 26 16:27:24.582765 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.582602    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76"} err="failed to get container status \"1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76\": rpc error: code = NotFound desc = could not find container \"1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76\": container with ID starting with 1db1e76acade3212560edd1296c40b40c997fd6f77612e67b4f572401927df76 not found: ID does not exist"
May 26 16:27:24.582765 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.582626    2570 scope.go:117] "RemoveContainer" containerID="d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6"
May 26 16:27:24.582936 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:27:24.582845    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6\": container with ID starting with d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6 not found: ID does not exist" containerID="d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6"
May 26 16:27:24.582936 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.582868    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6"} err="failed to get container status \"d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6\": rpc error: code = NotFound desc = could not find container \"d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6\": container with ID starting with d1a38c1f03a9a6c3d3c4604f4d3dca92bf82697a78c80655ea97974da651a2d6 not found: ID does not exist"
May 26 16:27:24.588877 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.588855    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"]
May 26 16:27:24.593189 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.593170    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4"]
May 26 16:27:24.625682 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.625656    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/168729a0-3ea9-4f06-a196-fec2df57a92c-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:27:24.625682 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.625681    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-a007a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/168729a0-3ea9-4f06-a196-fec2df57a92c-success-200-isvc-a007a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:27:24.625825 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:24.625691    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49s6x\" (UniqueName: \"kubernetes.io/projected/168729a0-3ea9-4f06-a196-fec2df57a92c-kube-api-access-49s6x\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:27:25.236507 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:25.236473    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" path="/var/lib/kubelet/pods/168729a0-3ea9-4f06-a196-fec2df57a92c/volumes"
May 26 16:27:25.309609 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:25.309564    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a007a-predictor-5f5fc59776-qcnp4" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
May 26 16:27:28.567142 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:28.567111    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:27:28.567628 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:28.567602    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 16:27:29.429012 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:29.428986    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:27:38.567764 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:38.567681    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 16:27:48.568599 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:48.568555    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 16:27:58.568017 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:27:58.567975    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 16:28:08.568944 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:28:08.568914    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:35:56.264298 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.264246    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"]
May 26 16:35:56.266868 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.264500    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container" containerID="cri-o://59fc08341bc4e94b614a7004b6db7012d99842f2da2d9695c10238fac56d0790" gracePeriod=30
May 26 16:35:56.266868 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.264536    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kube-rbac-proxy" containerID="cri-o://44b914c3f94b72a335dfb083f6f607f39c0463f6c55383b9ae3263d4e88e2163" gracePeriod=30
May 26 16:35:56.379227 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.379195    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"]
May 26 16:35:56.379662 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.379645    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kube-rbac-proxy"
May 26 16:35:56.379740 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.379665    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kube-rbac-proxy"
May 26 16:35:56.379740 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.379689    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container"
May 26 16:35:56.379740 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.379698    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container"
May 26 16:35:56.379903 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.379829    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kserve-container"
May 26 16:35:56.379903 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.379842    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="168729a0-3ea9-4f06-a196-fec2df57a92c" containerName="kube-rbac-proxy"
May 26 16:35:56.382971 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.382951    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.385257 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.385238    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-82fbc-predictor-serving-cert\""
May 26 16:35:56.385381 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.385303    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-82fbc-kube-rbac-proxy-sar-config\""
May 26 16:35:56.396624 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.396604    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"]
May 26 16:35:56.397823 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.397797    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-proxy-tls\") pod \"success-200-isvc-82fbc-predictor-76cd8c4d48-88z99\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") " pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.397932 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.397837    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qhs\" (UniqueName: \"kubernetes.io/projected/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-kube-api-access-b5qhs\") pod \"success-200-isvc-82fbc-predictor-76cd8c4d48-88z99\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") " pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.397984 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.397950    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-success-200-isvc-82fbc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-82fbc-predictor-76cd8c4d48-88z99\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") " pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.499304 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.499249    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-success-200-isvc-82fbc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-82fbc-predictor-76cd8c4d48-88z99\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") " pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.499479 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.499377    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-proxy-tls\") pod \"success-200-isvc-82fbc-predictor-76cd8c4d48-88z99\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") " pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.499479 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.499405    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qhs\" (UniqueName: \"kubernetes.io/projected/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-kube-api-access-b5qhs\") pod \"success-200-isvc-82fbc-predictor-76cd8c4d48-88z99\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") " pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.499875 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.499852    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-success-200-isvc-82fbc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-82fbc-predictor-76cd8c4d48-88z99\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") " pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.501678 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.501660    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-proxy-tls\") pod \"success-200-isvc-82fbc-predictor-76cd8c4d48-88z99\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") " pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.508514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.508492    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qhs\" (UniqueName: \"kubernetes.io/projected/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-kube-api-access-b5qhs\") pod \"success-200-isvc-82fbc-predictor-76cd8c4d48-88z99\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") " pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.693528 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.693491    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:56.831071 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.831043    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"]
May 26 16:35:56.832975 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:35:56.832949    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb675e0c3_2029_4f3e_9b8b_bdbec82b6161.slice/crio-40c36083ffb5131e737b0f21c3438f8940fc928775dda7f4ee230d0a0a9dc1cc WatchSource:0}: Error finding container 40c36083ffb5131e737b0f21c3438f8940fc928775dda7f4ee230d0a0a9dc1cc: Status 404 returned error can't find the container with id 40c36083ffb5131e737b0f21c3438f8940fc928775dda7f4ee230d0a0a9dc1cc
May 26 16:35:56.834586 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:56.834572    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:35:57.297925 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:57.297892    2570 generic.go:358] "Generic (PLEG): container finished" podID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerID="44b914c3f94b72a335dfb083f6f607f39c0463f6c55383b9ae3263d4e88e2163" exitCode=2
May 26 16:35:57.298365 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:57.297970    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" event={"ID":"dcd5a011-9458-4571-8981-038c5a4f73b7","Type":"ContainerDied","Data":"44b914c3f94b72a335dfb083f6f607f39c0463f6c55383b9ae3263d4e88e2163"}
May 26 16:35:57.299674 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:57.299652    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" event={"ID":"b675e0c3-2029-4f3e-9b8b-bdbec82b6161","Type":"ContainerStarted","Data":"62e998ddb8b9d4bfc9abedea2839b802a90f714f275256e95f06dbeb9672b1d0"}
May 26 16:35:57.299780 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:57.299679    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" event={"ID":"b675e0c3-2029-4f3e-9b8b-bdbec82b6161","Type":"ContainerStarted","Data":"72433fdcdd1d990c12556e0c661e2c9ecdd2a75cbffeb9f6d5f15ef7ebdbfbff"}
May 26 16:35:57.299780 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:57.299692    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" event={"ID":"b675e0c3-2029-4f3e-9b8b-bdbec82b6161","Type":"ContainerStarted","Data":"40c36083ffb5131e737b0f21c3438f8940fc928775dda7f4ee230d0a0a9dc1cc"}
May 26 16:35:57.299856 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:57.299811    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:57.318847 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:57.318808    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podStartSLOduration=1.31879598 podStartE2EDuration="1.31879598s" podCreationTimestamp="2026-05-26 16:35:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:35:57.317421448 +0000 UTC m=+2108.698743476" watchObservedRunningTime="2026-05-26 16:35:57.31879598 +0000 UTC m=+2108.700118004"
May 26 16:35:58.303043 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:58.303003    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:35:58.304168 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:58.304145    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 16:35:59.308432 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.308392    2570 generic.go:358] "Generic (PLEG): container finished" podID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerID="59fc08341bc4e94b614a7004b6db7012d99842f2da2d9695c10238fac56d0790" exitCode=0
May 26 16:35:59.308858 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.308476    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" event={"ID":"dcd5a011-9458-4571-8981-038c5a4f73b7","Type":"ContainerDied","Data":"59fc08341bc4e94b614a7004b6db7012d99842f2da2d9695c10238fac56d0790"}
May 26 16:35:59.308925 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.308859    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 16:35:59.420504 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.420483    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:35:59.530567 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.530498    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5a011-9458-4571-8981-038c5a4f73b7-success-200-isvc-2138e-kube-rbac-proxy-sar-config\") pod \"dcd5a011-9458-4571-8981-038c5a4f73b7\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") "
May 26 16:35:59.530567 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.530548    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5a011-9458-4571-8981-038c5a4f73b7-proxy-tls\") pod \"dcd5a011-9458-4571-8981-038c5a4f73b7\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") "
May 26 16:35:59.530769 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.530589    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx7b7\" (UniqueName: \"kubernetes.io/projected/dcd5a011-9458-4571-8981-038c5a4f73b7-kube-api-access-gx7b7\") pod \"dcd5a011-9458-4571-8981-038c5a4f73b7\" (UID: \"dcd5a011-9458-4571-8981-038c5a4f73b7\") "
May 26 16:35:59.530908 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.530885    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd5a011-9458-4571-8981-038c5a4f73b7-success-200-isvc-2138e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-2138e-kube-rbac-proxy-sar-config") pod "dcd5a011-9458-4571-8981-038c5a4f73b7" (UID: "dcd5a011-9458-4571-8981-038c5a4f73b7"). InnerVolumeSpecName "success-200-isvc-2138e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:35:59.532549 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.532514    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd5a011-9458-4571-8981-038c5a4f73b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcd5a011-9458-4571-8981-038c5a4f73b7" (UID: "dcd5a011-9458-4571-8981-038c5a4f73b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:35:59.532636 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.532607    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd5a011-9458-4571-8981-038c5a4f73b7-kube-api-access-gx7b7" (OuterVolumeSpecName: "kube-api-access-gx7b7") pod "dcd5a011-9458-4571-8981-038c5a4f73b7" (UID: "dcd5a011-9458-4571-8981-038c5a4f73b7"). InnerVolumeSpecName "kube-api-access-gx7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:35:59.632066 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.632035    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-2138e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcd5a011-9458-4571-8981-038c5a4f73b7-success-200-isvc-2138e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:35:59.632066 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.632060    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcd5a011-9458-4571-8981-038c5a4f73b7-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:35:59.632066 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:35:59.632070    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx7b7\" (UniqueName: \"kubernetes.io/projected/dcd5a011-9458-4571-8981-038c5a4f73b7-kube-api-access-gx7b7\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:36:00.313737 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:00.313699    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f" event={"ID":"dcd5a011-9458-4571-8981-038c5a4f73b7","Type":"ContainerDied","Data":"3ecdb088e8b6631461dfa6671e91e4400e7208c8e29ea38fa3db0c4e92285430"}
May 26 16:36:00.313737 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:00.313725    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"
May 26 16:36:00.314172 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:00.313747    2570 scope.go:117] "RemoveContainer" containerID="44b914c3f94b72a335dfb083f6f607f39c0463f6c55383b9ae3263d4e88e2163"
May 26 16:36:00.322001 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:00.321980    2570 scope.go:117] "RemoveContainer" containerID="59fc08341bc4e94b614a7004b6db7012d99842f2da2d9695c10238fac56d0790"
May 26 16:36:00.339315 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:00.339292    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"]
May 26 16:36:00.345319 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:00.345296    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2138e-predictor-5d4757c55b-zwp2f"]
May 26 16:36:01.236127 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:01.236081    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" path="/var/lib/kubelet/pods/dcd5a011-9458-4571-8981-038c5a4f73b7/volumes"
May 26 16:36:04.314753 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:04.314725    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:36:04.315367 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:04.315340    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 16:36:14.316268 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:14.316228    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 16:36:24.315831 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:24.315789    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 16:36:34.315452 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:34.315416    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
May 26 16:36:36.064697 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.064663    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"]
May 26 16:36:36.065153 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.065043    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container" containerID="cri-o://68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322" gracePeriod=30
May 26 16:36:36.065153 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.065111    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kube-rbac-proxy" containerID="cri-o://574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805" gracePeriod=30
May 26 16:36:36.069588 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.069561    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"]
May 26 16:36:36.070394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.069929    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kube-rbac-proxy"
May 26 16:36:36.070394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.069941    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kube-rbac-proxy"
May 26 16:36:36.070394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.069953    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container"
May 26 16:36:36.070394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.069959    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container"
May 26 16:36:36.070394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.070048    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kserve-container"
May 26 16:36:36.070394 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.070061    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="dcd5a011-9458-4571-8981-038c5a4f73b7" containerName="kube-rbac-proxy"
May 26 16:36:36.073545 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.073523    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.075664 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.075639    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bffff-predictor-serving-cert\""
May 26 16:36:36.075753 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.075711    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bffff-kube-rbac-proxy-sar-config\""
May 26 16:36:36.082362 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.082341    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"]
May 26 16:36:36.153327 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.153299    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-proxy-tls\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.153471 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.153347    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhh4c\" (UniqueName: \"kubernetes.io/projected/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-kube-api-access-bhh4c\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.153471 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.153376    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-success-200-isvc-bffff-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.254007 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.253980    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-proxy-tls\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.254194 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.254020    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bhh4c\" (UniqueName: \"kubernetes.io/projected/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-kube-api-access-bhh4c\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.254194 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.254041    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-success-200-isvc-bffff-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.254194 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:36:36.254132    2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-bffff-predictor-serving-cert: secret "success-200-isvc-bffff-predictor-serving-cert" not found
May 26 16:36:36.254401 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:36:36.254203    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-proxy-tls podName:cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2 nodeName:}" failed. No retries permitted until 2026-05-26 16:36:36.75418578 +0000 UTC m=+2148.135507782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-proxy-tls") pod "success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" (UID: "cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2") : secret "success-200-isvc-bffff-predictor-serving-cert" not found
May 26 16:36:36.254668 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.254649    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-success-200-isvc-bffff-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.272231 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.272210    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhh4c\" (UniqueName: \"kubernetes.io/projected/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-kube-api-access-bhh4c\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.449657 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.449624    2570 generic.go:358] "Generic (PLEG): container finished" podID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerID="574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805" exitCode=2
May 26 16:36:36.449858 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.449686    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" event={"ID":"fffa82fa-f5e1-4b1f-a032-1145a6df65f4","Type":"ContainerDied","Data":"574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805"}
May 26 16:36:36.759790 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.759702    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-proxy-tls\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.762047 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.762029    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-proxy-tls\") pod \"success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") " pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:36.988507 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:36.988463    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:37.120991 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:37.120682    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"]
May 26 16:36:37.123643 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:36:37.123614    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde33cb9_e03d_4fae_9ce5_565ba7e7c5d2.slice/crio-150471c737dfea7bc2b24524c975d9615284fe477f13d77f9c4f2e7ab016db08 WatchSource:0}: Error finding container 150471c737dfea7bc2b24524c975d9615284fe477f13d77f9c4f2e7ab016db08: Status 404 returned error can't find the container with id 150471c737dfea7bc2b24524c975d9615284fe477f13d77f9c4f2e7ab016db08
May 26 16:36:37.455069 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:37.455035    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" event={"ID":"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2","Type":"ContainerStarted","Data":"8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023"}
May 26 16:36:37.455069 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:37.455073    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" event={"ID":"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2","Type":"ContainerStarted","Data":"49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8"}
May 26 16:36:37.455290 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:37.455094    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:37.455290 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:37.455109    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" event={"ID":"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2","Type":"ContainerStarted","Data":"150471c737dfea7bc2b24524c975d9615284fe477f13d77f9c4f2e7ab016db08"}
May 26 16:36:37.455290 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:37.455122    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:37.456653 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:37.456628    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
May 26 16:36:37.472378 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:37.472331    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podStartSLOduration=1.472318453 podStartE2EDuration="1.472318453s" podCreationTimestamp="2026-05-26 16:36:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:36:37.471190152 +0000 UTC m=+2148.852512174" watchObservedRunningTime="2026-05-26 16:36:37.472318453 +0000 UTC m=+2148.853640477"
May 26 16:36:38.459915 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:38.459877    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
May 26 16:36:38.563055 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:38.562970    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
May 26 16:36:38.568312 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:38.568267    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 26 16:36:39.323812 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.323788    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:36:39.464757 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.464670    2570 generic.go:358] "Generic (PLEG): container finished" podID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerID="68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322" exitCode=0
May 26 16:36:39.464757 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.464724    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" event={"ID":"fffa82fa-f5e1-4b1f-a032-1145a6df65f4","Type":"ContainerDied","Data":"68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322"}
May 26 16:36:39.464757 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.464743    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"
May 26 16:36:39.464757 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.464756    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd" event={"ID":"fffa82fa-f5e1-4b1f-a032-1145a6df65f4","Type":"ContainerDied","Data":"0711b170ada4181c5cdddbd572aef98b2e24be857dea414f05369d966a02baa9"}
May 26 16:36:39.465329 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.464773    2570 scope.go:117] "RemoveContainer" containerID="574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805"
May 26 16:36:39.474988 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.474970    2570 scope.go:117] "RemoveContainer" containerID="68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322"
May 26 16:36:39.482359 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.482339    2570 scope.go:117] "RemoveContainer" containerID="574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805"
May 26 16:36:39.482659 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:36:39.482639    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805\": container with ID starting with 574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805 not found: ID does not exist" containerID="574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805"
May 26 16:36:39.482723 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.482668    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805"} err="failed to get container status \"574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805\": rpc error: code = NotFound desc = could not find container \"574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805\": container with ID starting with 574a5e0a6ef702d5827a02382038c4ee86d97154e7082a378d4b5fd16f833805 not found: ID does not exist"
May 26 16:36:39.482723 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.482687    2570 scope.go:117] "RemoveContainer" containerID="68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322"
May 26 16:36:39.482902 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:36:39.482888    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322\": container with ID starting with 68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322 not found: ID does not exist" containerID="68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322"
May 26 16:36:39.482944 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.482906    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322"} err="failed to get container status \"68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322\": rpc error: code = NotFound desc = could not find container \"68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322\": container with ID starting with 68f0646a850560a37a831a66675a2195beb26bf839a5a7b52b38053e44461322 not found: ID does not exist"
May 26 16:36:39.484143 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.484126    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-success-200-isvc-86373-kube-rbac-proxy-sar-config\") pod \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") "
May 26 16:36:39.484239 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.484227    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-proxy-tls\") pod \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") "
May 26 16:36:39.484304 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.484296    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cmq2\" (UniqueName: \"kubernetes.io/projected/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-kube-api-access-4cmq2\") pod \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\" (UID: \"fffa82fa-f5e1-4b1f-a032-1145a6df65f4\") "
May 26 16:36:39.484451 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.484434    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-success-200-isvc-86373-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-86373-kube-rbac-proxy-sar-config") pod "fffa82fa-f5e1-4b1f-a032-1145a6df65f4" (UID: "fffa82fa-f5e1-4b1f-a032-1145a6df65f4"). InnerVolumeSpecName "success-200-isvc-86373-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:36:39.484545 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.484523    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-86373-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-success-200-isvc-86373-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:36:39.486154 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.486130    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fffa82fa-f5e1-4b1f-a032-1145a6df65f4" (UID: "fffa82fa-f5e1-4b1f-a032-1145a6df65f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:36:39.486221 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.486160    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-kube-api-access-4cmq2" (OuterVolumeSpecName: "kube-api-access-4cmq2") pod "fffa82fa-f5e1-4b1f-a032-1145a6df65f4" (UID: "fffa82fa-f5e1-4b1f-a032-1145a6df65f4"). InnerVolumeSpecName "kube-api-access-4cmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:36:39.585362 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.585331    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cmq2\" (UniqueName: \"kubernetes.io/projected/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-kube-api-access-4cmq2\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:36:39.585362 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.585357    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fffa82fa-f5e1-4b1f-a032-1145a6df65f4-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:36:39.791142 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.791106    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"]
May 26 16:36:39.797154 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:39.797131    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86373-predictor-59f94db984-rq2zd"]
May 26 16:36:41.237110 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:41.237075    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" path="/var/lib/kubelet/pods/fffa82fa-f5e1-4b1f-a032-1145a6df65f4/volumes"
May 26 16:36:43.464397 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:43.464367    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:36:43.464996 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:43.464968    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
May 26 16:36:44.315879 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:44.315849    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:36:53.465256 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:36:53.465211    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
May 26 16:37:03.465542 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:03.465498    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
May 26 16:37:06.560795 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.560759    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"]
May 26 16:37:06.561254 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.561109    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container" containerID="cri-o://72433fdcdd1d990c12556e0c661e2c9ecdd2a75cbffeb9f6d5f15ef7ebdbfbff" gracePeriod=30
May 26 16:37:06.561254 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.561126    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kube-rbac-proxy" containerID="cri-o://62e998ddb8b9d4bfc9abedea2839b802a90f714f275256e95f06dbeb9672b1d0" gracePeriod=30
May 26 16:37:06.602408 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.602378    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"]
May 26 16:37:06.602842 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.602824    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container"
May 26 16:37:06.602842 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.602842    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container"
May 26 16:37:06.602974 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.602862    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kube-rbac-proxy"
May 26 16:37:06.602974 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.602871    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kube-rbac-proxy"
May 26 16:37:06.602974 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.602932    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kube-rbac-proxy"
May 26 16:37:06.602974 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.602949    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="fffa82fa-f5e1-4b1f-a032-1145a6df65f4" containerName="kserve-container"
May 26 16:37:06.606532 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.606512    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.609339 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.609312    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5798a-predictor-serving-cert\""
May 26 16:37:06.609443 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.609322    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5798a-kube-rbac-proxy-sar-config\""
May 26 16:37:06.615838 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.615617    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"]
May 26 16:37:06.707072 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.707042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb44v\" (UniqueName: \"kubernetes.io/projected/916c4746-0b07-4f1b-a54d-f5f8a961e19f-kube-api-access-bb44v\") pod \"success-200-isvc-5798a-predictor-7b4956db94-97vx2\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") " pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.707233 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.707105    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/916c4746-0b07-4f1b-a54d-f5f8a961e19f-proxy-tls\") pod \"success-200-isvc-5798a-predictor-7b4956db94-97vx2\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") " pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.707233 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.707148    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/916c4746-0b07-4f1b-a54d-f5f8a961e19f-success-200-isvc-5798a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5798a-predictor-7b4956db94-97vx2\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") " pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.807880 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.807843    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bb44v\" (UniqueName: \"kubernetes.io/projected/916c4746-0b07-4f1b-a54d-f5f8a961e19f-kube-api-access-bb44v\") pod \"success-200-isvc-5798a-predictor-7b4956db94-97vx2\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") " pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.808064 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.807907    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/916c4746-0b07-4f1b-a54d-f5f8a961e19f-proxy-tls\") pod \"success-200-isvc-5798a-predictor-7b4956db94-97vx2\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") " pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.808064 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.807969    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/916c4746-0b07-4f1b-a54d-f5f8a961e19f-success-200-isvc-5798a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5798a-predictor-7b4956db94-97vx2\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") " pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.808591 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.808558    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/916c4746-0b07-4f1b-a54d-f5f8a961e19f-success-200-isvc-5798a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5798a-predictor-7b4956db94-97vx2\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") " pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.810343 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.810294    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/916c4746-0b07-4f1b-a54d-f5f8a961e19f-proxy-tls\") pod \"success-200-isvc-5798a-predictor-7b4956db94-97vx2\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") " pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.819827 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.819774    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb44v\" (UniqueName: \"kubernetes.io/projected/916c4746-0b07-4f1b-a54d-f5f8a961e19f-kube-api-access-bb44v\") pod \"success-200-isvc-5798a-predictor-7b4956db94-97vx2\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") " pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:06.920199 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:06.920172    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:07.048139 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:07.048111    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"]
May 26 16:37:07.051225 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:37:07.051194    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916c4746_0b07_4f1b_a54d_f5f8a961e19f.slice/crio-b12ba365428ce250eec88dfc3e9fa508b9d2edf976e6868d8fde4c57743993ba WatchSource:0}: Error finding container b12ba365428ce250eec88dfc3e9fa508b9d2edf976e6868d8fde4c57743993ba: Status 404 returned error can't find the container with id b12ba365428ce250eec88dfc3e9fa508b9d2edf976e6868d8fde4c57743993ba
May 26 16:37:07.566565 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:07.566536    2570 generic.go:358] "Generic (PLEG): container finished" podID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerID="62e998ddb8b9d4bfc9abedea2839b802a90f714f275256e95f06dbeb9672b1d0" exitCode=2
May 26 16:37:07.566947 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:07.566607    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" event={"ID":"b675e0c3-2029-4f3e-9b8b-bdbec82b6161","Type":"ContainerDied","Data":"62e998ddb8b9d4bfc9abedea2839b802a90f714f275256e95f06dbeb9672b1d0"}
May 26 16:37:07.568103 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:07.568075    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" event={"ID":"916c4746-0b07-4f1b-a54d-f5f8a961e19f","Type":"ContainerStarted","Data":"551874234e71e4b932644ef5e15b57e7cb5d0d849d612a5d3977da0c5c1e7d60"}
May 26 16:37:07.568222 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:07.568110    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" event={"ID":"916c4746-0b07-4f1b-a54d-f5f8a961e19f","Type":"ContainerStarted","Data":"0fde1f04243cf1c3124590316f271bb31d6ad7da11e19ffe313e53c39796f40a"}
May 26 16:37:07.568222 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:07.568124    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" event={"ID":"916c4746-0b07-4f1b-a54d-f5f8a961e19f","Type":"ContainerStarted","Data":"b12ba365428ce250eec88dfc3e9fa508b9d2edf976e6868d8fde4c57743993ba"}
May 26 16:37:07.568222 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:07.568212    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:07.592934 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:07.592893    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" podStartSLOduration=1.5928771099999999 podStartE2EDuration="1.59287711s" podCreationTimestamp="2026-05-26 16:37:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:37:07.591411129 +0000 UTC m=+2178.972733154" watchObservedRunningTime="2026-05-26 16:37:07.59287711 +0000 UTC m=+2178.974199136"
May 26 16:37:08.572093 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:08.572060    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:08.573196 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:08.573167    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
May 26 16:37:09.309242 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.309201    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
May 26 16:37:09.576715 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.576686    2570 generic.go:358] "Generic (PLEG): container finished" podID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerID="72433fdcdd1d990c12556e0c661e2c9ecdd2a75cbffeb9f6d5f15ef7ebdbfbff" exitCode=0
May 26 16:37:09.577079 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.576754    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" event={"ID":"b675e0c3-2029-4f3e-9b8b-bdbec82b6161","Type":"ContainerDied","Data":"72433fdcdd1d990c12556e0c661e2c9ecdd2a75cbffeb9f6d5f15ef7ebdbfbff"}
May 26 16:37:09.577324 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.577270    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
May 26 16:37:09.711489 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.711463    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:37:09.834024 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.833933    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5qhs\" (UniqueName: \"kubernetes.io/projected/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-kube-api-access-b5qhs\") pod \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") "
May 26 16:37:09.834205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.834030    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-success-200-isvc-82fbc-kube-rbac-proxy-sar-config\") pod \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") "
May 26 16:37:09.834205 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.834089    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-proxy-tls\") pod \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\" (UID: \"b675e0c3-2029-4f3e-9b8b-bdbec82b6161\") "
May 26 16:37:09.834467 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.834442    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-success-200-isvc-82fbc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-82fbc-kube-rbac-proxy-sar-config") pod "b675e0c3-2029-4f3e-9b8b-bdbec82b6161" (UID: "b675e0c3-2029-4f3e-9b8b-bdbec82b6161"). InnerVolumeSpecName "success-200-isvc-82fbc-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:37:09.836094 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.836072    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b675e0c3-2029-4f3e-9b8b-bdbec82b6161" (UID: "b675e0c3-2029-4f3e-9b8b-bdbec82b6161"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:37:09.836172 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.836108    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-kube-api-access-b5qhs" (OuterVolumeSpecName: "kube-api-access-b5qhs") pod "b675e0c3-2029-4f3e-9b8b-bdbec82b6161" (UID: "b675e0c3-2029-4f3e-9b8b-bdbec82b6161"). InnerVolumeSpecName "kube-api-access-b5qhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:37:09.935126 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.935093    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-82fbc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-success-200-isvc-82fbc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:37:09.935126 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.935123    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:37:09.935347 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:09.935136    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5qhs\" (UniqueName: \"kubernetes.io/projected/b675e0c3-2029-4f3e-9b8b-bdbec82b6161-kube-api-access-b5qhs\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:37:10.581161 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:10.581127    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99" event={"ID":"b675e0c3-2029-4f3e-9b8b-bdbec82b6161","Type":"ContainerDied","Data":"40c36083ffb5131e737b0f21c3438f8940fc928775dda7f4ee230d0a0a9dc1cc"}
May 26 16:37:10.581161 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:10.581148    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"
May 26 16:37:10.581637 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:10.581178    2570 scope.go:117] "RemoveContainer" containerID="62e998ddb8b9d4bfc9abedea2839b802a90f714f275256e95f06dbeb9672b1d0"
May 26 16:37:10.590554 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:10.590538    2570 scope.go:117] "RemoveContainer" containerID="72433fdcdd1d990c12556e0c661e2c9ecdd2a75cbffeb9f6d5f15ef7ebdbfbff"
May 26 16:37:10.610729 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:10.610705    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"]
May 26 16:37:10.615547 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:10.615526    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-82fbc-predictor-76cd8c4d48-88z99"]
May 26 16:37:11.237504 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:11.237469    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" path="/var/lib/kubelet/pods/b675e0c3-2029-4f3e-9b8b-bdbec82b6161/volumes"
May 26 16:37:13.465022 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:13.464983    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
May 26 16:37:14.581843 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:14.581809    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:37:14.582373 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:14.582348    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
May 26 16:37:23.465693 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:23.465664    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:37:24.582769 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:24.582731    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
May 26 16:37:34.582839 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:34.582798    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
May 26 16:37:44.582461 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:44.582418    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
May 26 16:37:54.583332 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:37:54.583298    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:46:21.412484 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:21.412451    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"]
May 26 16:46:21.414928 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:21.412729    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container" containerID="cri-o://0fde1f04243cf1c3124590316f271bb31d6ad7da11e19ffe313e53c39796f40a" gracePeriod=30
May 26 16:46:21.414928 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:21.412780    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kube-rbac-proxy" containerID="cri-o://551874234e71e4b932644ef5e15b57e7cb5d0d849d612a5d3977da0c5c1e7d60" gracePeriod=30
May 26 16:46:22.451182 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:22.451149    2570 generic.go:358] "Generic (PLEG): container finished" podID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerID="551874234e71e4b932644ef5e15b57e7cb5d0d849d612a5d3977da0c5c1e7d60" exitCode=2
May 26 16:46:22.451584 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:22.451227    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" event={"ID":"916c4746-0b07-4f1b-a54d-f5f8a961e19f","Type":"ContainerDied","Data":"551874234e71e4b932644ef5e15b57e7cb5d0d849d612a5d3977da0c5c1e7d60"}
May 26 16:46:24.459848 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.459810    2570 generic.go:358] "Generic (PLEG): container finished" podID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerID="0fde1f04243cf1c3124590316f271bb31d6ad7da11e19ffe313e53c39796f40a" exitCode=0
May 26 16:46:24.460149 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.459880    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" event={"ID":"916c4746-0b07-4f1b-a54d-f5f8a961e19f","Type":"ContainerDied","Data":"0fde1f04243cf1c3124590316f271bb31d6ad7da11e19ffe313e53c39796f40a"}
May 26 16:46:24.460149 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.459914    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2" event={"ID":"916c4746-0b07-4f1b-a54d-f5f8a961e19f","Type":"ContainerDied","Data":"b12ba365428ce250eec88dfc3e9fa508b9d2edf976e6868d8fde4c57743993ba"}
May 26 16:46:24.460149 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.459923    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b12ba365428ce250eec88dfc3e9fa508b9d2edf976e6868d8fde4c57743993ba"
May 26 16:46:24.463596 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.463578    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:46:24.552550 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.552468    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/916c4746-0b07-4f1b-a54d-f5f8a961e19f-proxy-tls\") pod \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") "
May 26 16:46:24.552550 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.552508    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb44v\" (UniqueName: \"kubernetes.io/projected/916c4746-0b07-4f1b-a54d-f5f8a961e19f-kube-api-access-bb44v\") pod \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") "
May 26 16:46:24.552550 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.552544    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/916c4746-0b07-4f1b-a54d-f5f8a961e19f-success-200-isvc-5798a-kube-rbac-proxy-sar-config\") pod \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\" (UID: \"916c4746-0b07-4f1b-a54d-f5f8a961e19f\") "
May 26 16:46:24.552942 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.552915    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916c4746-0b07-4f1b-a54d-f5f8a961e19f-success-200-isvc-5798a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-5798a-kube-rbac-proxy-sar-config") pod "916c4746-0b07-4f1b-a54d-f5f8a961e19f" (UID: "916c4746-0b07-4f1b-a54d-f5f8a961e19f"). InnerVolumeSpecName "success-200-isvc-5798a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:46:24.554542 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.554507    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916c4746-0b07-4f1b-a54d-f5f8a961e19f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "916c4746-0b07-4f1b-a54d-f5f8a961e19f" (UID: "916c4746-0b07-4f1b-a54d-f5f8a961e19f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:46:24.554626 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.554581    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916c4746-0b07-4f1b-a54d-f5f8a961e19f-kube-api-access-bb44v" (OuterVolumeSpecName: "kube-api-access-bb44v") pod "916c4746-0b07-4f1b-a54d-f5f8a961e19f" (UID: "916c4746-0b07-4f1b-a54d-f5f8a961e19f"). InnerVolumeSpecName "kube-api-access-bb44v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:46:24.653528 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.653494    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/916c4746-0b07-4f1b-a54d-f5f8a961e19f-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:46:24.653528 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.653524    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb44v\" (UniqueName: \"kubernetes.io/projected/916c4746-0b07-4f1b-a54d-f5f8a961e19f-kube-api-access-bb44v\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:46:24.653528 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:24.653535    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-5798a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/916c4746-0b07-4f1b-a54d-f5f8a961e19f-success-200-isvc-5798a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:46:25.463137 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:25.463101    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"
May 26 16:46:25.480598 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:25.480570    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"]
May 26 16:46:25.484402 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:25.484378    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5798a-predictor-7b4956db94-97vx2"]
May 26 16:46:27.237055 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:27.237013    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" path="/var/lib/kubelet/pods/916c4746-0b07-4f1b-a54d-f5f8a961e19f/volumes"
May 26 16:46:49.404376 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:49.404347    2570 scope.go:117] "RemoveContainer" containerID="551874234e71e4b932644ef5e15b57e7cb5d0d849d612a5d3977da0c5c1e7d60"
May 26 16:46:49.412496 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:46:49.412475    2570 scope.go:117] "RemoveContainer" containerID="0fde1f04243cf1c3124590316f271bb31d6ad7da11e19ffe313e53c39796f40a"
May 26 16:53:55.512187 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:55.512155    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"]
May 26 16:53:55.514709 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:55.512441    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container" containerID="cri-o://49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8" gracePeriod=30
May 26 16:53:55.514709 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:55.512488    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kube-rbac-proxy" containerID="cri-o://8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023" gracePeriod=30
May 26 16:53:56.014583 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:56.014544    2570 generic.go:358] "Generic (PLEG): container finished" podID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerID="8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023" exitCode=2
May 26 16:53:56.014583 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:56.014584    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" event={"ID":"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2","Type":"ContainerDied","Data":"8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023"}
May 26 16:53:58.460492 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.460447    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
May 26 16:53:58.563005 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.562981    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:53:58.723980 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.723944    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhh4c\" (UniqueName: \"kubernetes.io/projected/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-kube-api-access-bhh4c\") pod \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") "
May 26 16:53:58.723980 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.723986    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-success-200-isvc-bffff-kube-rbac-proxy-sar-config\") pod \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") "
May 26 16:53:58.724203 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.724064    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-proxy-tls\") pod \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\" (UID: \"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2\") "
May 26 16:53:58.724422 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.724378    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-success-200-isvc-bffff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-bffff-kube-rbac-proxy-sar-config") pod "cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" (UID: "cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2"). InnerVolumeSpecName "success-200-isvc-bffff-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:53:58.726071 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.726050    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-kube-api-access-bhh4c" (OuterVolumeSpecName: "kube-api-access-bhh4c") pod "cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" (UID: "cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2"). InnerVolumeSpecName "kube-api-access-bhh4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:53:58.726184 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.726161    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" (UID: "cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:53:58.825514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.825469    2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-proxy-tls\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:53:58.825514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.825508    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhh4c\" (UniqueName: \"kubernetes.io/projected/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-kube-api-access-bhh4c\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:53:58.825514 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:58.825521    2570 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-bffff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2-success-200-isvc-bffff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-89.ec2.internal\" DevicePath \"\""
May 26 16:53:59.025713 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.025623    2570 generic.go:358] "Generic (PLEG): container finished" podID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerID="49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8" exitCode=0
May 26 16:53:59.025713 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.025704    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"
May 26 16:53:59.025884 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.025709    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" event={"ID":"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2","Type":"ContainerDied","Data":"49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8"}
May 26 16:53:59.025884 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.025749    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj" event={"ID":"cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2","Type":"ContainerDied","Data":"150471c737dfea7bc2b24524c975d9615284fe477f13d77f9c4f2e7ab016db08"}
May 26 16:53:59.025884 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.025764    2570 scope.go:117] "RemoveContainer" containerID="8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023"
May 26 16:53:59.035195 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.035172    2570 scope.go:117] "RemoveContainer" containerID="49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8"
May 26 16:53:59.042466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.042447    2570 scope.go:117] "RemoveContainer" containerID="8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023"
May 26 16:53:59.042718 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:53:59.042697    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023\": container with ID starting with 8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023 not found: ID does not exist" containerID="8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023"
May 26 16:53:59.042783 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.042725    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023"} err="failed to get container status \"8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023\": rpc error: code = NotFound desc = could not find container \"8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023\": container with ID starting with 8c3e4bd73ee1d17478e27ce75f00622ca91cbf59aea37e20613bc2193b97b023 not found: ID does not exist"
May 26 16:53:59.042783 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.042745    2570 scope.go:117] "RemoveContainer" containerID="49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8"
May 26 16:53:59.042966 ip-10-0-129-89 kubenswrapper[2570]: E0526 16:53:59.042947    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8\": container with ID starting with 49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8 not found: ID does not exist" containerID="49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8"
May 26 16:53:59.043006 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.042969    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8"} err="failed to get container status \"49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8\": rpc error: code = NotFound desc = could not find container \"49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8\": container with ID starting with 49b301b1299c144d68f943344f5b94b3e7c714900efc0af71ebc38aa4eb25fa8 not found: ID does not exist"
May 26 16:53:59.048908 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.048883    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"]
May 26 16:53:59.052914 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.052894    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bffff-predictor-6cc9bbd78-2lwkj"]
May 26 16:53:59.237093 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:53:59.237059    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" path="/var/lib/kubelet/pods/cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2/volumes"
May 26 16:54:30.227391 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:30.227360    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jqhkk_799f1c88-c483-4ec4-b68c-4b9e2850bcc8/global-pull-secret-syncer/0.log"
May 26 16:54:30.286137 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:30.286105    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kbdlx_9185f8d9-5823-4f14-bfb4-7d311127a7f1/konnectivity-agent/0.log"
May 26 16:54:30.378449 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:30.378421    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-89.ec2.internal_84135cd3e968802e4af704bc56619b8f/haproxy/0.log"
May 26 16:54:33.743021 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.742995    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8/alertmanager/0.log"
May 26 16:54:33.767623 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.767597    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8/config-reloader/0.log"
May 26 16:54:33.789147 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.789123    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8/kube-rbac-proxy-web/0.log"
May 26 16:54:33.811152 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.811132    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8/kube-rbac-proxy/0.log"
May 26 16:54:33.832999 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.832968    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8/kube-rbac-proxy-metric/0.log"
May 26 16:54:33.854016 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.853997    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8/prom-label-proxy/0.log"
May 26 16:54:33.877813 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.877781    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d63d95a3-fa42-4bc0-bb47-f7453a9e4ec8/init-config-reloader/0.log"
May 26 16:54:33.954032 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.954005    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-dlzz7_41f76d72-50c2-441c-9a2c-e2214d8fe34e/kube-state-metrics/0.log"
May 26 16:54:33.974206 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.974130    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-dlzz7_41f76d72-50c2-441c-9a2c-e2214d8fe34e/kube-rbac-proxy-main/0.log"
May 26 16:54:33.996161 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:33.996140    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-dlzz7_41f76d72-50c2-441c-9a2c-e2214d8fe34e/kube-rbac-proxy-self/0.log"
May 26 16:54:34.046306 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:34.046271    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7888fb4b49-lfbg2_bc101f86-414b-4376-9851-b0668457cdd8/metrics-server/0.log"
May 26 16:54:34.075189 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:34.075166    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-tqjc9_66bc4d8e-4730-4cc9-97d5-48fb27980a67/monitoring-plugin/0.log"
May 26 16:54:34.208864 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:34.208825    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvcn8_db87ceaf-fcba-4bfa-9e4d-3a448a6f9224/node-exporter/0.log"
May 26 16:54:34.238670 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:34.238601    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvcn8_db87ceaf-fcba-4bfa-9e4d-3a448a6f9224/kube-rbac-proxy/0.log"
May 26 16:54:34.260755 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:34.260731    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvcn8_db87ceaf-fcba-4bfa-9e4d-3a448a6f9224/init-textfile/0.log"
May 26 16:54:34.736003 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:34.735974    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c4b68ff49-d95c8_a07153bc-afe6-4068-bec7-b741121f5c09/telemeter-client/0.log"
May 26 16:54:34.770234 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:34.770208    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c4b68ff49-d95c8_a07153bc-afe6-4068-bec7-b741121f5c09/reload/0.log"
May 26 16:54:34.797535 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:34.797508    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c4b68ff49-d95c8_a07153bc-afe6-4068-bec7-b741121f5c09/kube-rbac-proxy/0.log"
May 26 16:54:36.939256 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:36.939228    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7586f45f75-hth66_53e8a5bc-366b-4b32-911c-0adaaa222933/console/0.log"
May 26 16:54:36.978813 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:36.978778    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-xmq5t_6f1529c1-8dd5-4612-9e01-8a453e28f70a/download-server/0.log"
May 26 16:54:37.304988 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.304906    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"]
May 26 16:54:37.305311 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305296    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kube-rbac-proxy"
May 26 16:54:37.305368 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305313    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kube-rbac-proxy"
May 26 16:54:37.305368 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305332    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container"
May 26 16:54:37.305368 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305338    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container"
May 26 16:54:37.305368 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305345    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kube-rbac-proxy"
May 26 16:54:37.305368 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305351    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kube-rbac-proxy"
May 26 16:54:37.305368 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305356    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kube-rbac-proxy"
May 26 16:54:37.305368 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305361    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kube-rbac-proxy"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305371    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305378    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305387    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305394    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305443    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kube-rbac-proxy"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305450    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kube-rbac-proxy"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305461    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kube-rbac-proxy"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305470    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="916c4746-0b07-4f1b-a54d-f5f8a961e19f" containerName="kserve-container"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305477    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="cde33cb9-e03d-4fae-9ce5-565ba7e7c5d2" containerName="kserve-container"
May 26 16:54:37.305618 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.305485    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="b675e0c3-2029-4f3e-9b8b-bdbec82b6161" containerName="kserve-container"
May 26 16:54:37.307508 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.307493    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.309655 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.309627    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sh5kf\"/\"kube-root-ca.crt\""
May 26 16:54:37.309789 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.309631    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sh5kf\"/\"openshift-service-ca.crt\""
May 26 16:54:37.310473 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.310457    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sh5kf\"/\"default-dockercfg-hlrkd\""
May 26 16:54:37.317208 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.317185    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"]
May 26 16:54:37.466208 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.466171    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-lib-modules\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.466417 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.466223    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xx2p\" (UniqueName: \"kubernetes.io/projected/8675fbda-6115-4c93-a745-6c7c0dfc2b04-kube-api-access-6xx2p\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.466417 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.466249    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-proc\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.466417 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.466309    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-podres\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.466417 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.466330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-sys\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.567150 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.567062    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-sys\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.567150 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.567139    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-lib-modules\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.567399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.567177    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-sys\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.567399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.567189    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6xx2p\" (UniqueName: \"kubernetes.io/projected/8675fbda-6115-4c93-a745-6c7c0dfc2b04-kube-api-access-6xx2p\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.567399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.567229    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-proc\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.567399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.567271    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-podres\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.567399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.567336    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-lib-modules\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.567399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.567357    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-proc\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.567399 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.567389    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8675fbda-6115-4c93-a745-6c7c0dfc2b04-podres\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.575714 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.575685    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xx2p\" (UniqueName: \"kubernetes.io/projected/8675fbda-6115-4c93-a745-6c7c0dfc2b04-kube-api-access-6xx2p\") pod \"perf-node-gather-daemonset-rn9mj\" (UID: \"8675fbda-6115-4c93-a745-6c7c0dfc2b04\") " pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.617453 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.617422    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:37.742739 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.742702    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"]
May 26 16:54:37.746891 ip-10-0-129-89 kubenswrapper[2570]: W0526 16:54:37.746864    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8675fbda_6115_4c93_a745_6c7c0dfc2b04.slice/crio-3aeb3732d23fa7272bc37f5d3ad8c775a5c0303cd5b7b072e10c8ecdfb6bb2b4 WatchSource:0}: Error finding container 3aeb3732d23fa7272bc37f5d3ad8c775a5c0303cd5b7b072e10c8ecdfb6bb2b4: Status 404 returned error can't find the container with id 3aeb3732d23fa7272bc37f5d3ad8c775a5c0303cd5b7b072e10c8ecdfb6bb2b4
May 26 16:54:37.748831 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:37.748813    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:54:38.118561 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:38.118483    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2lbjs_9ea83ed2-100d-4de2-a94d-5aed66a964a7/dns/0.log"
May 26 16:54:38.140621 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:38.140595    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2lbjs_9ea83ed2-100d-4de2-a94d-5aed66a964a7/kube-rbac-proxy/0.log"
May 26 16:54:38.162834 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:38.162807    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj" event={"ID":"8675fbda-6115-4c93-a745-6c7c0dfc2b04","Type":"ContainerStarted","Data":"41a19cd4923685b341fd4297433879f2c9e77b08d1b87608466527daa7f55715"}
May 26 16:54:38.162993 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:38.162839    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj" event={"ID":"8675fbda-6115-4c93-a745-6c7c0dfc2b04","Type":"ContainerStarted","Data":"3aeb3732d23fa7272bc37f5d3ad8c775a5c0303cd5b7b072e10c8ecdfb6bb2b4"}
May 26 16:54:38.162993 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:38.162927    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:38.179882 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:38.179834    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj" podStartSLOduration=1.179820429 podStartE2EDuration="1.179820429s" podCreationTimestamp="2026-05-26 16:54:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:54:38.17838909 +0000 UTC m=+3229.559711115" watchObservedRunningTime="2026-05-26 16:54:38.179820429 +0000 UTC m=+3229.561142457"
May 26 16:54:38.249165 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:38.249140    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qm4f5_e3f547af-2cf4-4821-ae28-e2087a814658/dns-node-resolver/0.log"
May 26 16:54:38.718883 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:38.718803    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78b4b769ff-tfc2v_4b2cfe75-3d68-4a25-971d-0b9c4b2be4b6/registry/0.log"
May 26 16:54:38.744967 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:38.744939    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mm79x_bda7246b-9f6c-4339-86e9-2a97c2370213/node-ca/0.log"
May 26 16:54:39.488908 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:39.488870    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f76bdfcbb-7twj9_b0ad9be8-9903-47e5-bc99-2c61845cce5e/router/0.log"
May 26 16:54:39.887006 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:39.886976    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xd8kr_a560b45b-1006-463e-8922-5b41684d6458/serve-healthcheck-canary/0.log"
May 26 16:54:40.273393 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:40.273251    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-5nqbd_6fe2ee2b-83d4-401a-a6e1-389caeeb7e09/insights-operator/0.log"
May 26 16:54:40.274466 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:40.274448    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-5nqbd_6fe2ee2b-83d4-401a-a6e1-389caeeb7e09/insights-operator/1.log"
May 26 16:54:40.293062 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:40.293036    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-564hz_c3267091-97bf-4e59-b36b-ae7157a89f87/kube-rbac-proxy/0.log"
May 26 16:54:40.314153 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:40.314117    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-564hz_c3267091-97bf-4e59-b36b-ae7157a89f87/exporter/0.log"
May 26 16:54:40.336442 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:40.336414    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-564hz_c3267091-97bf-4e59-b36b-ae7157a89f87/extractor/0.log"
May 26 16:54:42.398404 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:42.398368    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b94ff949c-9dqgq_cdb1320c-d47a-4e45-b0c3-1953a35cb52e/manager/0.log"
May 26 16:54:42.865538 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:42.865460    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-6nklx_c1c8d171-8b19-4e7e-804d-e0e8fda9f101/manager/0.log"
May 26 16:54:42.922981 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:42.922957    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-mmjxf_00d6a42d-2d5f-465c-ab2f-0195f5ddbc0c/seaweedfs/0.log"
May 26 16:54:44.176794 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:44.176769    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sh5kf/perf-node-gather-daemonset-rn9mj"
May 26 16:54:48.241838 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.241814    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pjqf_59beaf4f-19bd-4f7c-be73-07dad7f9fde2/kube-multus-additional-cni-plugins/0.log"
May 26 16:54:48.262433 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.262357    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pjqf_59beaf4f-19bd-4f7c-be73-07dad7f9fde2/egress-router-binary-copy/0.log"
May 26 16:54:48.285038 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.285015    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pjqf_59beaf4f-19bd-4f7c-be73-07dad7f9fde2/cni-plugins/0.log"
May 26 16:54:48.307405 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.307384    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pjqf_59beaf4f-19bd-4f7c-be73-07dad7f9fde2/bond-cni-plugin/0.log"
May 26 16:54:48.330856 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.330832    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pjqf_59beaf4f-19bd-4f7c-be73-07dad7f9fde2/routeoverride-cni/0.log"
May 26 16:54:48.351346 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.351316    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pjqf_59beaf4f-19bd-4f7c-be73-07dad7f9fde2/whereabouts-cni-bincopy/0.log"
May 26 16:54:48.372232 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.372210    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pjqf_59beaf4f-19bd-4f7c-be73-07dad7f9fde2/whereabouts-cni/0.log"
May 26 16:54:48.749758 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.749720    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k7bjv_be3af2fc-3d53-46f3-b681-d665b14c11a8/kube-multus/0.log"
May 26 16:54:48.884578 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.884471    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r88dc_02034fc7-cc11-48cf-8eff-ae2384a2080e/network-metrics-daemon/0.log"
May 26 16:54:48.904401 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:48.904367    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r88dc_02034fc7-cc11-48cf-8eff-ae2384a2080e/kube-rbac-proxy/0.log"
May 26 16:54:49.660766 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:49.660737    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8fv6s_270dc9ab-90fe-4504-8313-236f5a761696/ovn-controller/0.log"
May 26 16:54:49.706138 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:49.706104    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8fv6s_270dc9ab-90fe-4504-8313-236f5a761696/ovn-acl-logging/0.log"
May 26 16:54:49.728300 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:49.728190    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8fv6s_270dc9ab-90fe-4504-8313-236f5a761696/kube-rbac-proxy-node/0.log"
May 26 16:54:49.751055 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:49.751028    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8fv6s_270dc9ab-90fe-4504-8313-236f5a761696/kube-rbac-proxy-ovn-metrics/0.log"
May 26 16:54:49.768161 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:49.768130    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8fv6s_270dc9ab-90fe-4504-8313-236f5a761696/northd/0.log"
May 26 16:54:49.792932 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:49.792906    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8fv6s_270dc9ab-90fe-4504-8313-236f5a761696/nbdb/0.log"
May 26 16:54:49.816696 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:49.816671    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8fv6s_270dc9ab-90fe-4504-8313-236f5a761696/sbdb/0.log"
May 26 16:54:49.995731 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:49.995650    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8fv6s_270dc9ab-90fe-4504-8313-236f5a761696/ovnkube-controller/0.log"
May 26 16:54:51.615379 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:51.615348    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-xgtzm_ce52bd45-0e01-4b65-814b-294a61999d19/check-endpoints/0.log"
May 26 16:54:51.638092 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:51.638065    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-968nq_49b37583-51bd-40e5-94c6-e1b191d0a891/network-check-target-container/0.log"
May 26 16:54:52.536862 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:52.536831    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-27bnz_557f56c9-c491-4f35-99de-92a9cdc50f23/iptables-alerter/0.log"
May 26 16:54:53.186475 ip-10-0-129-89 kubenswrapper[2570]: I0526 16:54:53.186395    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-62bbh_ddfb6a13-33b0-4ea4-ac38-ec92b912632e/tuned/0.log"